1.1 ABSTRACT:
Big sensor data is prevalent in both industry and scientific research applications where the data is generated with high volume and velocity it is difficult to process using on-hand database management tools or traditional data processing applications. Cloud computing provides a promising platform to support the addressing of this challenge as it provides a flexible stack of massive computing, storage, and software services in a scalable manner at low cost. Some techniques have been developed in recent years for processing sensor data on cloud, such as sensor-cloud. However, these techniques do not provide efficient support on fast detection and locating of errors in big sensor data sets.
We develop a novel data error detection approach which exploits the full computation potential of cloud platform and the network feature of WSN. Firstly, a set of sensor data error types are classified and defined. Based on that classification, the network feature of a clustered WSN is introduced and analyzed to support fast error detection and location. Specifically, in our proposed approach, the error detection is based on the scale-free network topology and most of detection operations can be conducted in limited temporal or spatial data blocks instead of a whole big data set. Hence the detection and location process can be dramatically accelerated.
Furthermore, the detection and location
tasks can be distributed to cloud platform to fully exploit the computation
power and massive storage. Through the experiment on our cloud computing
platform of U-Cloud, it is demonstrated that our proposed approach can
significantly reduce the time for error detection and location in big data sets
generated by large scale sensor network systems with acceptable error detecting
accuracy.
1.2 INTRODUCTION:
Recently, we enter a new era of data explosion which brings about new challenges for big data processing. In general, big data is a collection of data sets so large and complex that it becomes difficult to process with onhand database management systems or traditional data processing applications. It represents the progress of the human cognitive processes, usually includes data sets with sizes beyond the ability of current technology, method and theory to capture, manage, and process the data within a tolerable elapsed time. Big data has typical characteristics of five ‘V’s, volume, variety, velocity, veracity and value. Big data sets come from many areas, including meteorology, connectomics, complex physics simulations, genomics, biological study, gene analysis and environmental research. According to literature since 1980s, generated data doubles its size in every 40 months all over the world. In the year of 2012, there were 2.5 quintillion (2.5 1018) bytes of data being generated every day.
Hence, how to process big data has become a
fundamental and critical challenge for modern society. Cloud computing provides
apromising platform for big data processing with powerful computation
capability, storage, scalability, resource reuse and low cost, and has
attracted significant attention in alignment with big data. One of important
source for scientific big data is the data sets collected by wireless sensor
networks (WSN). Wireless sensor networks have potential of significantly
enhancing people’s ability to monitor and interact with their physical
environment. Big data set from sensors is often subject to corruption and
losses due to wireless medium of communication and presence of hardware
inaccuracies in the nodes. For a WSN application to deduce an appropriate
result, it is necessary that the data received is clean, accurate, and
lossless. However, effective detection and cleaning of sensor big data errors
is a challenging issue demanding innovative solutions. WSN with cloud can be categorized
as a kind of complex network systems. In these complex network systems such as
WSN and social network, data abnormality and error become an annoying issue for
the real network applications.
Therefore, the question of how to find data errors in complex network systems for improving and debugging the network has attracted the interests of researchers. Some work has been done for big data analysis and error detection in complex networks including intelligence sensors networks. There are also some works related to complex network systems data error detection and debugging with online data processing techniques. Since these techniques were not designed and developed to deal with big data on cloud, they were unable to cope with current dramatic increase of data size. For example, when big data sets are encountered, previous offline methods for error detectionand debugging on a single computer may take a long time and lose real time feedback. Because those offline methods are normally based on learning or mining, they often introduce high time cost during the process of data set training and pattern matching. WSN big data error detection commonly requires powerful real-time processing and storing of the massive sensor data as well as analysis in the context of using inherently complex error models to identify and locate events of abnormalities.
In this paper, we aim to develop a novel error detection approach by exploiting the massive storage, scalability and computation power of cloud to detect errors in big data sets from sensor networks. Some work has been done about processing sensor data on cloud. However, fast detection of data errors in big data with cloud remains challenging. Especially, how to use the computation power of cloud to quickly find and locate errors of nodes in WSN needs to be explored. Cloud computing, a disruptive trend at present, poses a significant impact on current IT industry and research communities. Cloud computing infrastructure is becoming popular because it provides an open, flexible, scalable and reconfigurable platform. The proposed error detection approach in this paper will be based on the classification of error types. Specifically, nine types of numerical data abnormalities/errors are listed and introduced in our cloud error detection approach. The defined error model will trigger the error detection process. Compared to previous error detection of sensor network systems, our approach on cloud will be designed and developed by utilizing the massive data processing capability of cloud to enhance error detection speed and real time reaction. In addition, the architecture feature of complex networks will also be analyzed to combine with the cloud computing with a more efficient way. Based on current research literature review, we divide complex network systems into scale-free type and non scale-free type. Sensor network is a kind of scale-free complex network system which matches cloud scalability feature.
1.3 LITRATURE SURVEY
A SURVEY OF LARGE SCALE DATA MANAGEMENT APPROACHES IN CLOUD ENVIRONMENTS
PUBLISH: IEEE Comm. Surveys & Tutorials, vol. 13, no. 3, pp. 311-336, Third Quarter 2011.
AUTHOR: S. Sakr, A. Liu, D. Batista, and M. Alomari,
EXPLANATION:
In the last two decades, the
continuous increase of computational power has produced an overwhelming flow of
data. Moreover, the recent advances in Web technology has made it easy for any
user to provide and consume content of any form. This has called for a paradigm
shift in the computing architecture and large scale data processing mechanisms.
Cloud computing is associated with a new paradigm for the provision of
computing infrastructure. This paradigm shifts the location of this
infrastructure to the network to reduce the costs associated with the
management of hardware and software resources. This paper gives a comprehensive
survey of numerous approaches and mechanisms of deploying data-intensive
applications in the cloud which are gaining a lot of momentum in both research
and industrial communities. We analyze the various design decisions of each
approach and its suitability to support certain classes of applications and
end-users. A discussion of some open issues and future challenges pertaining to
scalability, consistency, economical processing of large scale data on the
cloud is provided. We highlight the characteristics of the best candidate
classes of applications that can be deployed in the cloud.
STREAM AS YOU GO: THE CASE FOR INCREMENTAL DATA ACCESS AND PROCESSING IN THE CLOUD
PUBLISH: Proc. IEEE ICDE Int’l Workshop Data Management in the Cloud (DMC’12), 2012.
AUTHOR: R. Kienzler, R. Bruggmann, A. Ranganathan, and N. Tatbul,
EXPLANATION:
Cloud infrastructures promise to
provide high-performance and cost-effective solutions to large-scale data
processing problems. In this paper, we identify a common class of
data-intensive applications for which data transfer latency for uploading data
into the cloud in advance of its processing may hinder the linear scalability
advantage of the cloud. For such applications, we propose a
“stream-as-you-go” approach for incrementally accessing and
processing data based on a stream data management architecture. We describe our
approach in the context of a DNA sequence analysis use case and compare it
against the state of the art in MapReduce-based DNA sequence analysis and
incremental MapReduce frameworks. We provide experimental results over an
implementation of our approach based on the IBM InfoSphere Streams computing
platform deployed on Amazon EC2, showing an order of magnitude improvement in
total processing time over the state of the art.
A SCALABLE TWO-PHASE TOP-DOWN SPECIALIZATION APPROACH FOR DATA ANONYMIZATION USING SYSTEMS, IN MAPREDUCE ON CLOUD
PUBLISH: IEEE Trans. Parallel and Distributed, vol. 25, no. 2, pp. 363-373, Feb. 2014.
AUTHOR: X. Zhang, T. Yang, C. Liu, and J. Chen
EXPLANATION:
A large number of cloud services
require users to share private data like electronic health records for data
analysis or mining, bringing privacy concerns. Anonymizing data sets via
generalization to satisfy certain privacy requirements such as k-anonymity is a
widely used category of privacy preserving techniques. At present, the scale of
data in many cloud applications increases tremendously in accordance with the
Big Data trend, thereby making it a challenge for commonly used software tools
to capture, manage, and process such large-scale data within a tolerable
elapsed time. As a result, it is a challenge for existing anonymization
approaches to achieve privacy preservation on privacy-sensitive large-scale
data sets due to their insufficiency of scalability. In this paper, we propose
a scalable two-phase top-down specialization (TDS) approach to anonymize
large-scale data sets using the MapReduce framework on cloud. In both phases of
our approach, we deliberately design a group of innovative MapReduce jobs to
concretely accomplish the specialization computation in a highly scalable way.
Experimental evaluation results demonstrate that with our approach, the
scalability and efficiency of TDS can be significantly improved over existing
approaches.
CHAPTER 2
2.0 SYSTEM ANALYSIS
2.1 EXISTING SYSTEM:
A data error in big data with cloud remains challenging to use the computation power of cloud to quickly find and locate errors of nodes in WSN needs to be explored. Cloud computing, a disruptive trend at present, poses a significant impact on current IT industry and research communities. Cloud computing infrastructure is becoming popular because it provides an open, flexible, scalable and reconfigurable platform. Existing methods in wireless sensor networks is to provide low-cost, low-energy reliable data collection. Reliability against transient errors in sensor data can be provided using the model-based error correction described in which temporal correlation in the data is used to correct errors without any overheads at the sensor nodes. In the above work it is assumed that a perfect model of the data is available.
However, as variations
in the physical process are context-dependent and time-varying in a real sensor
network, it is infeasible to have an accurate model of the data properties a
priori, thus leading to reduced correction efficiency issue by presenting a
scalable methodology for improving the accuracy of data modeling through
on-line estimation data correction algorithm to incorporate robustness against
dynamic model changes and potential modeling errors. We evaluate our system
through simulations using real sensor data collected from different sources.
Experimental results demonstrate that the proposed enhancements lead to an
improvement of up to a factor of 10 over the earlier approach.
2.1.1 DISADVANTAGES:
Ensuring the reliability of sensor data
becomes harder, since the hardware becomes less robust to many types of errors
due to the effects of aggressive technology scaling. Similarly, errors in the
wireless communication channels are another source of unreliability, as
limitations on transmission power due to tight energy constraints makes them
more susceptible to noise and interference. The problem is further aggravated
by exposure to harsh physical environments, which is common for many typical
sensing applications. Subsequently, ensuring the reliability of the data in a
sensor network is going to be a growing problem and be a challenging part of
designing sensor networks.
2.2 PROPOSED SYSTEM:
We proposed error detection approach in this paper will be based on the classification of error types. Specifically, nine types of numerical data abnormalities/errors are listed and introduced in our cloud error detection approach. The defined error model will trigger the error detection process. Compared to previous error detection of sensor network systems, our approach on cloud will be designed and developed by utilizing the massive data processing capability of cloud to enhance error detection speed and real time reaction. However, the scalability and error detection accuracy are not dealt. It is an initial and important step for online error detection of WSN.
Especially, under the cloud environment, the computational power and scalability should be fully exploit to support the real time fast error detection for sensor data sets clustering can significantly reduce the time cost error locating and final decision making by avoiding whole network data processing. In addition, with this detection technique, cloud resources only need be distributed according to each partitioned cluster in a scale-free complex network on current research literature review, we divide complex network systems into scale-free type and non scale-free type. Sensor network is a kind of scale-free complex network system which matches cloud scalability feature.
Our proposed error detection approach on cloud is specifically trimmed for finding errors in big data sets of sensor networks. The main contribution of our proposed detection is to achieve significant time performance improvement in error detection without compromising error detection accuracy. Our proposed scale-free error detection algorithm achieves significant error detection performance gains compared to non scale-free error detection algorithms. Our proposed scale-free detection on cloud can fast detect most of error data (more than 80 percent) after 740 seconds time duration. However, the non scalefree error detection algorithm can only achieve as much as 44 percent error detection rate as the best case. So, it can be concluded from the experiment results in Fig. 5 that the scale-free detection algorithm on cloud for big data can significantly outperform non scale-free error detection algorithms in terms of error finding time cost.
2.2.1 ADVANTAGES:
To verify the time efficiency and the effectiveness of our approach for detecting errors in big data with cloud, experiments are conducted for this experiment.
2.3.1 HARDWARE REQUIREMENT:
CHAPTER 3
3.0 SYSTEM DESIGN:
Data Flow Diagram / Use Case Diagram / Flow Diagram:
External sources or destinations, which may be people or organizations or other entities
Here the data referenced by a process is stored and retrieved.
People, procedures or devices that produce data’s in the physical component is not identified.
Data moves in a specific direction from an origin to a destination. The data flow is a “packet” of data.
MODELING RULES:
There are several common modeling rules when creating DFDs:
3.1 ARCHITECTURE DIAGRAM
3.2 DATAFLOW DIAGRAM:
UML DIAGRAMS:
3.2 USE CASE DIAGRAM:
START RESULTS
3.3 CLASS DIAGRAM:
3.4 SEQUENCE DIAGRAM:
STRAT RESULTS
Data Structure
Cluster Analysis
Complexity Analysis
Using Error Detection Algorithm
Error Localization
Classification and Complexity Analysis
Results View Graph
3.5 ACTIVITY DIAGRAM:
CHAPTER 4
4.0 IMPLEMENTATION:
MODEL BASED ERROR DETECTION ON CLOUD FOR SENSOR NETWORK BIG DATA
ERROR DETECTION:
We propose a two-phase approach to conduct the computation required in the whole process of error detection and localization. At the phase of error detection, there are three inputs for the error detection algorithm. The first is the graph of network. The second is the total collected data set D and the third is the defined error patterns p. The output of the error detection algorithm is the error set D’. The details of the error detection algorithm can be found in Appendix B.1, available in the online supplemental material.
ERROR LOCALIZATION:
After the error pattern matching and
error detection, it is important to locate the position and source of the
detected error in the original WSN graph G(V, E). The input of the Algorithm 2
is the original graph of a scale-free network G (V, E), and an error data D
from Algorithm 1. The output of the algorithm 2 is G’(V’, E’) which is the
subset of the G to indicate the error location and source. The details of the
error detection algorithm can be found in Appendix B.2, available in the online
supplemental material.
COMPLEXITY ANALYSIS:
Suppose that there is a sensor network system consisting of n nodes. For the error detection approach without considering the scale-free network feature, the error detection algorithm will carry out the error pattern matching and localization with whole network data by traversing the whole data set. Suppose that there is R nodes on the data routing, in the worst case, the detection algorithm without considering the scale-free network feature will be executed R n time for error detection and localization, denoted as OðR nÞ; 1 R n. Anyway, with the hierarchical network topology, the network can be partitioned in to m clusters.
Model based on our scale-free network
definition and our algorithm, in each cluster, the nodes which are involved in
error detection will be reduced to n/m on average. In addition, in each
cluster, the data values are highly correlated. The data worst case of data
traverse times for error detection and localization is determined. Because our
scale-free error detection approach limits most of computation within each
cluster, the communication and data exchange between clusters can be ignored.
Finally, the worst case algorithm complexity of our scale-free error detection
approach can outperform the traditional error detection algorithms.
4.1 ALGORITHM
Introduce the big data error detection/location algorithm, and its combination strategy with cloud. Our proposed algorithm on cloud, the data sets need to be partitioned before feeding to the algorithm on cloud. There are two points should be mentioned when carrying out partitioning. Firstly, the partition process could not bring new data errors into a data set; or change and influence the original errors in a data set. That is different to the previous partition algorithm which normally divides data set according certain application preference or clustering principles. Secondly, due to the scale-free network systems being a special topology, the partition has to form the data clusters according to the real world situation of scale-free network or Cluster-head based WSN.
MapReduce is a framework for processing parallelizable problems across huge data sets using a large number of computers (nodes), collectively referred to as a cluster (if all nodes are on the same local network and use similar hardware) or a grid (if the nodes are shared across geographically and administratively distributed systems, and use more heterogenous hardware). Computational processing can occur on data stored either in a filesystem (unstructured) or in a database (structured). MapReduce can take advantage of locality of data, processing data on or near the storage assets to reduce data transmission. “Map” function.
The master node takes the input, divides it into smaller subproblems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node. “Reduce” function. The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve. MapReduce allows for distributed processing of the map and reduction operations.
4.2 MODULES:
NETWORK TOPOLOGY DESIGNS:
ON-CLOUD PROCESSING FOR WSN:
TIME-EFFICIENT ERROR DETECTION:
ERROR AND ABNORMALITY CLASSIFICATION:
ERROR
DEFINITION AND MODELING:
4.3 MODULE DESCRIPTION:
NETWORK TOPOLOGY DESIGNS:
Scale-free networks are inhomogeneous and only a few nodes have a large number of links. In real applications, the cluster-head WSN is similar to scale-free networks, which can be described with the scale-free complex networks and has the feature of scale-free networks. In Fig. 2, the instance of scale-free networks and exponential networks are compared. It can be concluded that the scale-free networks have a more clustered hierarchical nodes topology. Central nodes are highly connected by the out-layer nodes has only 1 or 2 links the traditional error detection for WSN data sets has not paid enough attention to making use of complex network features to improve the error detection efficiency on the cloud platform. Compared to the previous sensor data error detection and localization approach, complex network topology features will be explored with the computation power of cloud for error detection efficiency, scalability and low cost.
Wireless sensor network systems have
been used in different areas, such as environment monitoring, military,
disaster warning and scientific data collection. In order to process the remote
sensor data collected by WSN, sensor-cloud platform has been developed
including its definition, architecture, and applications. Due to the features of
high variety, volume, and velocity, big data is difficult to process using
onhand database management tools or traditional sensorcloud platform. Big data
sets can come from complex network systems, such as social network and large
scale sensor networks. In addition, under the theme of complex network systems,
it may be difficult to develop timeefficient detecting or trouble-shooting
methods for errors in big data sets, hence to debug the complex network systems
in real time.
ON-CLOUD PROCESSING FOR WSN:
Sensor-Cloud is a unique sensor data storage, visualization and remote management platform that leverages powerful cloud computing technologies to provide excellent data scalability, fast visualization, and user programmable analysis. Initially, sensor-cloud was designed to support long-term deployments of MicroStrain wireless sensors. But nowadays, sensor-cloud has been developed to support any web-connected third party device, sensor, or sensor network through a simple OpenData API. Sensor-Cloud can be useful for a variety of applications, particularly where data from large sensor networks needs to be collected, viewed, and monitored remotely. For example, structural health monitoring and condition-based monitoring of high value assets are applications where commonly available data tools often come up short in terms of accessibility, data scalability, programmability, or performance.
Sensor-Cloud represents a direction for
processing and analyzing big sensor data using cloud platform. The online WSN
data quality and data cleaning issues are discussed in deal with the problems
of outliers, missing information, and noise. A novel online approach for
modeling and online learning of temporal-spatial data correlations in sensor
networks is developed. A Bayesian approach for reducing the effect of noise on
sensor data online is also proposed [37]. The proposed approach is efficient in
reducing the uncertainty associated with noisy sensors. However, the
scalability and error detection accuracy are not dealt. It is an initial and
important step for online error detection of WSN. But lots of work still needs
to be done. Especially, under the cloud environment, the computational power
and scalability should be fully exploit to support the real time fast error detection
for sensor data sets.
TIME-EFFICIENT ERROR DETECTION:
In this section, a cluster-head WSN will be introduced and processed as a kind of complex network system. These complex networks may have non-trivial statistical properties which will influence the data processing strategy on them. In order to test the false positive ratio of our error detection approach and time cost for error findings, we impose five types of data errors following the definition in Section 3 into the normalized testing data sets with a uniform random distribution. These five types of data errors are generated equally. Hence, the percentage of each type of errors is 20 percent from the total imposed errors for testing. The first imposed error type is the flat line error. The second imposed error type is out of bound error. The third imposed error type is the spike error. The forth imposed error type is the data lost error. Finally, the aggregate & fusion error type is imposed. By imposing the above listed five types of data error types, the experiment is designed to measure the error selection efficiency and accuracy during the on-cloud processing of data set.
Specifically, 10 different error rates
are imposed into the experimental data set and tested independently. The testing
error rate changes from 1 to 10 percent in 10 repetitive experiments. After
about 100 seconds, the proposed algorithm can detect more than 60 percent
errors whatever the testing error rate is within the domain between 1 and 10
percent . During the time duration between 0 and 100 second, all error
detection rates increase dramatically with a steep trend. After the time point
of 300 second, the error detection rates increase slowly with a flat trend. At
the time of 740 second, the proposed error detection algorithm on cloud can
find and locate more than 95 percent imposed errors from the testing data sets.
When testing error rate is 1 percent, the best performance gains are achieved,
as about 99.5 percent total errors detection. With the increase of the testing
error rate, the error detection rate decreases.
ERROR AND ABNORMALITY CLASSIFICATION:
Big data sets from real world complex networks, there are mainly two types of data generated and exchanged within networks. (1) The numeric data sampled and exchanged between network nodes such as sensor network sampled data sets. (2) The text files and data logs generated by nodes such as social network data sets. In this paper, our research will focus on the error detection for numeric big data sets from complex networks can be classified as six main types for both numeric and text data as Appendix A.1, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety. org/10.1109/ TPDS.2013.2295810. This error classification can effectively describe the common error types in complex network systems.
However, when it comes to the errors in wireless sensor network data sets, the above classification loses the accuracy in separating node or edge data error caused by different wireless data communication failures. In addition, it is not enough in describing the error data phenomena in sensor data sets. To better capture the error features of sensor data sets, the above general error classification in should be extended. Considering the specific feature of numeric data errors, there are several abnormal data scenarios demonstrated in Fig. 1. The “flat line faults” indicates a time series of a node in a network system keeps unchanged for unacceptable long time duration. In real world applications, sampled data and transmitted data always have slight changes with the time flow. The “out of data bounds faults” indicates impossible data values are observed based on some domain knowledge. In real world applications, if a temperature value of water is reported as 300
C, it can be treated as a data fault directly. The “data lost
fault” means there are missing data values in a time series during the data
generation or communication.
ERROR DEFINITION AND MODELING:
With the above classification, the definition of each error type is presented to guide our error detection algorithm. Suppose that a data record from a network node is denoted as r(n, t, f(n, t), g(n, l)), where n is the ID of the node in a network systems. t represents the window length of a time series. f(n, t) is the numerical values collected within window t from the node n. g(n, l) is a location function which records the cluster, the data source node and partition situation related to the node n. g(n, l) is used to calculate the distance between the data source node n and the node l which is the initial data source node. g(n, l) indicates that a current detected error data node is the initial data source node. Furthermore, g(n, l) is also used to parse the data routing between data communication nodes.
CHAPTER 5
5.0 SYSTEM STUDY:
5.1 FEASIBILITY STUDY:
The feasibility of the project is analyzed in this phase and business proposal is put forth with a very general plan for the project and some cost estimates. During system analysis the feasibility study of the proposed system is to be carried out. This is to ensure that the proposed system is not a burden to the company. For feasibility analysis, some understanding of the major requirements for the system is essential.
Three key considerations involved in the feasibility analysis are
5.1.1 ECONOMICAL FEASIBILITY:
This study is carried out to check the economic impact that the system will have on the organization. The amount of fund that the company can pour into the research and development of the system is limited. The expenditures must be justified. Thus the developed system as well within the budget and this was achieved because most of the technologies used are freely available. Only the customized products had to be purchased.
This study is carried out to check the technical feasibility, that is, the technical requirements of the system. Any system developed must not have a high demand on the available technical resources. This will lead to high demands on the available technical resources. This will lead to high demands being placed on the client. The developed system must have a modest requirement, as only minimal or null changes are required for implementing this system.
5.1.3 SOCIAL FEASIBILITY:
The aspect of study is to check the level of acceptance of the system by the user. This includes the process of training the user to use the system efficiently. The user must not feel threatened by the system, instead must accept it as a necessity. The level of acceptance by the users solely depends on the methods that are employed to educate the user about the system and to make him familiar with it. His level of confidence must be raised so that he is also able to make some constructive criticism, which is welcomed, as he is the final user of the system.
5.2 SYSTEM TESTING:
Testing is a process of checking whether the developed system is working according to the original objectives and requirements. It is a set of activities that can be planned in advance and conducted systematically. Testing is vital to the success of the system. System testing makes a logical assumption that if all the parts of the system are correct, the global will be successfully achieved. In adequate testing if not testing leads to errors that may not appear even many months.
This creates two problems, the time lag between the cause and the appearance of the problem and the effect of the system errors on the files and records within the system. A small system error can conceivably explode into a much larger Problem. Effective testing early in the purpose translates directly into long term cost savings from a reduced number of errors. Another reason for system testing is its utility, as a user-oriented vehicle before implementation. The best programs are worthless if it produces the correct outputs.
5.2.1 UNIT TESTING:
| Description | Expected result |
| Test for application window properties. | All the properties of the windows are to be properly aligned and displayed. |
| Test for mouse operations. | All the mouse operations like click, drag, etc. must perform the necessary operations without any exceptions. |
A program represents the
logical elements of a system. For a program to run satisfactorily, it must
compile and test data correctly and tie in properly with other programs.
Achieving an error free program is the responsibility of the programmer.
Program testing checks
for two types
of errors: syntax
and logical. Syntax error is a
program statement that violates one or more rules of the language in which it
is written. An improperly defined field dimension or omitted keywords are
common syntax errors. These errors are shown through error message generated by
the computer. For Logic errors the programmer must examine the output
carefully.
5.1.2 FUNCTIONAL TESTING:
Functional testing of an application is used to prove the application delivers correct results, using enough inputs to give an adequate level of confidence that will work correctly for all sets of inputs. The functional testing will need to prove that the application works for each client type and that personalization function work correctly.When a program is tested, the actual output is compared with the expected output. When there is a discrepancy the sequence of instructions must be traced to determine the problem. The process is facilitated by breaking the program into self-contained portions, each of which can be checked at certain key points. The idea is to compare program values against desk-calculated values to isolate the problems.
| Description | Expected result |
| Test for all modules. | All peers should communicate in the group. |
| Test for various peer in a distributed network framework as it display all users available in the group. | The result after execution should give the accurate result. |
5.1. 3 NON-FUNCTIONAL TESTING:
The Non Functional software testing encompasses a rich spectrum of testing strategies, describing the expected results for every test case. It uses symbolic analysis techniques. This testing used to check that an application will work in the operational environment. Non-functional testing includes:
5.1.4 LOAD TESTING:
An important tool for implementing system tests is a Load generator. A Load generator is essential for testing quality requirements such as performance and stress. A load can be a real load, that is, the system can be put under test to real usage by having actual telephone users connected to it. They will generate test input data for system test.
| Description | Expected result |
| It is necessary to ascertain that the application behaves correctly under loads when ‘Server busy’ response is received. | Should designate another active node as a Server. |
5.1.5 PERFORMANCE TESTING:
Performance tests are utilized in order to determine the widely defined performance of the software system such as execution time associated with various parts of the code, response time and device utilization. The intent of this testing is to identify weak points of the software system and quantify its shortcomings.
| Description | Expected result |
| This is required to assure that an application perforce adequately, having the capability to handle many peers, delivering its results in expected time and using an acceptable level of resource and it is an aspect of operational management. | Should handle large input values, and produce accurate result in a expected time. |
5.1.6 RELIABILITY TESTING:
The software reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time and it is being ensured in this testing. Reliability can be expressed as the ability of the software to reveal defects under testing conditions, according to the specified requirements. It the portability that a software system will operate without failure under given conditions for a given time interval and it focuses on the behavior of the software element. It forms a part of the software quality control team.
| Description | Expected result |
| This is to check that the server is rugged and reliable and can handle the failure of any of the components involved in provide the application. | In case of failure of the server an alternate server should take over the job. |
5.1.7 SECURITY TESTING:
Security testing evaluates system characteristics that relate to the availability, integrity and confidentiality of the system data and services. Users/Clients should be encouraged to make sure their security needs are very clearly known at requirements time, so that the security issues can be addressed by the designers and testers.
| Description | Expected result |
| Checking that the user identification is authenticated. | In case failure it should not be connected in the framework. |
| Check whether group keys in a tree are shared by all peers. | The peers should know group key in the same group. |
5.1.8 WHITE BOX TESTING:
White box testing, sometimes called glass-box testing is a test case design method that uses the control structure of the procedural design to derive test cases. Using white box testing method, the software engineer can derive test cases. The White box testing focuses on the inner structure of the software structure to be tested.
| Description | Expected result |
| Exercise all logical decisions on their true and false sides. | All the logical decisions must be valid. |
| Execute all loops at their boundaries and within their operational bounds. | All the loops must be finite. |
| Exercise internal data structures to ensure their validity. | All the data structures must be valid. |
5.1.9 BLACK BOX TESTING:
Black box testing, also called behavioral testing, focuses on the functional requirements of the software. That is, black testing enables the software engineer to derive sets of input conditions that will fully exercise all functional requirements for a program. Black box testing is not alternative to white box techniques. Rather it is a complementary approach that is likely to uncover a different class of errors than white box methods. Black box testing attempts to find errors which focuses on inputs, outputs, and principle function of a software module. The starting point of the black box testing is either a specification or code. The contents of the box are hidden and the stimulated software should produce the desired results.
| Description | Expected result |
| To check for incorrect or missing functions. | All the functions must be valid. |
| To check for interface errors. | The entire interface must function normally. |
| To check for errors in a data structures or external data base access. | The database updation and retrieval must be done. |
| To check for initialization and termination errors. | All the functions and data structures must be initialized properly and terminated normally. |
All
the above system testing strategies are carried out in as the development,
documentation and institutionalization of the proposed goals and related
policies is essential.
CHAPTER 6
6.0 SOFTWARE DESCRIPTION:
Java technology is both a programming language and a platform.
With most programming languages, you either compile or interpret a program so that you can run it on your computer. The Java programming language is unusual in that a program is both compiled and interpreted. With the compiler, first you translate a program into an intermediate language called Java byte codes —the platform-independent codes interpreted by the interpreter on the Java platform. The interpreter parses and runs each Java byte code instruction on the computer. Compilation happens just once; interpretation occurs each time the program is executed. The following figure illustrates how this works.
You can think of Java byte codes as the machine code instructions for the Java Virtual Machine (Java VM). Every Java interpreter, whether it’s a development tool or a Web browser that can run applets, is an implementation of the Java VM. Java byte codes help make “write once, run anywhere” possible. You can compile your program into byte codes on any platform that has a Java compiler. The byte codes can then be run on any implementation of the Java VM. That means that as long as a computer has a Java VM, the same program written in the Java programming language can run on Windows 2000, a Solaris workstation, or on an iMac.
A platform is the hardware or software environment in which a program runs. We’ve already mentioned some of the most popular platforms like Windows 2000, Linux, Solaris, and MacOS. Most platforms can be described as a combination of the operating system and hardware. The Java platform differs from most other platforms in that it’s a software-only platform that runs on top of other hardware-based platforms.
The Java platform has two components:
You’ve already been introduced to the Java VM. It’s the base for the Java platform and is ported onto various hardware-based platforms.
The Java API is a large collection of ready-made software components that provide many useful capabilities, such as graphical user interface (GUI) widgets. The Java API is grouped into libraries of related classes and interfaces; these libraries are known as packages. The next section, What Can Java Technology Do? Highlights what functionality some of the packages in the Java API provide.
The following figure depicts a program that’s running on the Java platform. As the figure shows, the Java API and the virtual machine insulate the program from the hardware.
Native code is code that after you compile it, the compiled code runs on a specific hardware platform. As a platform-independent environment, the Java platform can be a bit slower than native code. However, smart compilers, well-tuned interpreters, and just-in-time byte code compilers can bring performance close to that of native code without threatening portability.
The most common types of programs written in the Java programming language are applets and applications. If you’ve surfed the Web, you’re probably already familiar with applets. An applet is a program that adheres to certain conventions that allow it to run within a Java-enabled browser.
However, the Java programming language is not just for writing cute, entertaining applets for the Web. The general-purpose, high-level Java programming language is also a powerful software platform. Using the generous API, you can write many types of programs.
An application is a standalone program that runs directly on the Java platform. A special kind of application known as a server serves and supports clients on a network. Examples of servers are Web servers, proxy servers, mail servers, and print servers. Another specialized program is a servlet.
A servlet can almost be thought of as an applet that runs on the server side. Java Servlets are a popular choice for building interactive web applications, replacing the use of CGI scripts. Servlets are similar to applets in that they are runtime extensions of applications. Instead of working in browsers, though, servlets run within Java Web servers, configuring or tailoring the server.
How does the API support all these kinds of programs? It does so with packages of software components that provides a wide range of functionality. Every full implementation of the Java platform gives you the following features:
The Java platform also has APIs for 2D and 3D graphics, accessibility, servers, collaboration, telephony, speech, animation, and more. The following figure depicts what is included in the Java 2 SDK.
We can’t promise you fame, fortune, or even a job if you learn the Java programming language. Still, it is likely to make your programs better and requires less effort than other languages. We believe that Java technology will help you do the following:
Microsoft Open Database Connectivity (ODBC) is a standard programming interface for application developers and database systems providers. Before ODBC became a de facto standard for Windows programs to interface with database systems, programmers had to use proprietary languages for each database they wanted to connect to. Now, ODBC has made the choice of the database system almost irrelevant from a coding perspective, which is as it should be. Application developers have much more important things to worry about than the syntax that is needed to port their program from one database to another when business needs suddenly change.
Through the ODBC Administrator in Control Panel, you can specify the particular database that is associated with a data source that an ODBC application program is written to use. Think of an ODBC data source as a door with a name on it. Each door will lead you to a particular database. For example, the data source named Sales Figures might be a SQL Server database, whereas the Accounts Payable data source could refer to an Access database. The physical database referred to by a data source can reside anywhere on the LAN.
The ODBC system files are not installed on your system by Windows 95. Rather, they are installed when you setup a separate database application, such as SQL Server Client or Visual Basic 4.0. When the ODBC icon is installed in Control Panel, it uses a file called ODBCINST.DLL. It is also possible to administer your ODBC data sources through a stand-alone program called ODBCADM.EXE. There is a 16-bit and a 32-bit version of this program and each maintains a separate list of ODBC data sources.
From a programming perspective, the beauty of ODBC is that the application can be written to use the same set of function calls to interface with any data source, regardless of the database vendor. The source code of the application doesn’t change whether it talks to Oracle or SQL Server. We only mention these two as an example. There are ODBC drivers available for several dozen popular database systems. Even Excel spreadsheets and plain text files can be turned into data sources. The operating system uses the Registry information written by ODBC Administrator to determine which low-level ODBC drivers are needed to talk to the data source (such as the interface to Oracle or SQL Server). The loading of the ODBC drivers is transparent to the ODBC application program. In a client/server environment, the ODBC API even handles many of the network issues for the application programmer.
The advantages
of this scheme are so numerous that you are probably thinking there must be
some catch. The only disadvantage of ODBC is that it isn’t as efficient as
talking directly to the native database interface. ODBC has had many detractors
make the charge that it is too slow. Microsoft has always claimed that the
critical factor in performance is the quality of the driver software that is
used. In our humble opinion, this is true. The availability of good ODBC
drivers has improved a great deal recently. And anyway, the criticism about
performance is somewhat analogous to those who said that compilers would never
match the speed of pure assembly language. Maybe not, but the compiler (or
ODBC) gives you the opportunity to write cleaner programs, which means you
finish sooner. Meanwhile, computers get faster every year.
6.6 JDBC:
In an effort to set an independent database standard API for Java; Sun Microsystems developed Java Database Connectivity, or JDBC. JDBC offers a generic SQL database access mechanism that provides a consistent interface to a variety of RDBMSs. This consistent interface is achieved through the use of “plug-in” database connectivity modules, or drivers. If a database vendor wishes to have JDBC support, he or she must provide the driver for each platform that the database and Java run on.
To gain a wider acceptance of JDBC, Sun based JDBC’s framework on ODBC. As you discovered earlier in this chapter, ODBC has widespread support on a variety of platforms. Basing JDBC on ODBC will allow vendors to bring JDBC drivers to market much faster than developing a completely new connectivity solution.
JDBC was announced in March of 1996. It was released for a 90 day public review that ended June 8, 1996. Because of user input, the final JDBC v1.0 specification was released soon after.
The remainder of this section will cover enough information about JDBC for you to know what it is about and how to use it effectively. This is by no means a complete overview of JDBC. That would fill an entire book.
Few software packages are designed without goals in mind. JDBC is one that, because of its many goals, drove the development of the API. These goals, in conjunction with early reviewer feedback, have finalized the JDBC class library into a solid framework for building database applications in Java.
The goals that were set for JDBC are important. They will give you some insight as to why certain classes and functionalities behave the way they do. The eight design goals for JDBC are as follows:
SQL Level API
The designers felt that their main goal was to define a SQL interface for Java. Although not the lowest database interface level possible, it is at a low enough level for higher-level tools and APIs to be created. Conversely, it is at a high enough level for application programmers to use it confidently. Attaining this goal allows for future tool vendors to “generate” JDBC code and to hide many of JDBC’s complexities from the end user.
SQL Conformance
SQL syntax varies as you move from database vendor to database vendor. In an effort to support a wide variety of vendors, JDBC will allow any query statement to be passed through it to the underlying database driver. This allows the connectivity module to handle non-standard functionality in a manner that is suitable for its users.
JDBC must be implemental on top of common database interfaces
The JDBC SQL API must “sit” on top of other common SQL level APIs. This goal allows JDBC to use existing ODBC level drivers by the use of a software interface. This interface would translate JDBC calls to ODBC and vice versa.
Because of Java’s acceptance in the user community thus far, the designers feel that they should not stray from the current design of the core Java system.
This goal probably appears in all software design goal listings. JDBC is no exception. Sun felt that the design of JDBC should be very simple, allowing for only one method of completing a task per mechanism. Allowing duplicate functionality only serves to confuse the users of the API.
Strong typing allows for more error checking to be done at compile time; also, less error appear at runtime.
Because more often than not, the usual SQL calls
used by the programmer are simple SELECT’s,
INSERT’s,
DELETE’s
and UPDATE’s,
these queries should be simple to perform with JDBC. However, more complex SQL
statements should also be possible.
Finally we decided to precede the implementation using Java Networking.
And for dynamically updating the cache table we go for MS Access database.
Java ha two things: a programming language and a platform.
Java is a high-level programming language that is all of the following
Simple Architecture-neutral
Object-oriented Portable
Distributed High-performance
Interpreted Multithreaded
Robust Dynamic Secure
Java is also unusual in that each Java program is both compiled and interpreted. With a compile you translate a Java program into an intermediate language called Java byte codes the platform-independent code instruction is passed and run on the computer.
Compilation happens just once; interpretation occurs each time the program is executed. The figure illustrates how this works.
The TCP/IP stack is shorter than the OSI one:
TCP is a connection-oriented protocol; UDP (User Datagram Protocol) is a connectionless protocol.
The IP layer provides a connectionless and unreliable delivery system. It considers each datagram independently of the others. Any association between datagram must be supplied by the higher layers. The IP layer supplies a checksum that includes its own header. The header includes the source and destination addresses. The IP layer handles routing through an Internet. It is also responsible for breaking up large datagram into smaller ones for transmission and reassembling them at the other end.
UDP is also connectionless and unreliable. What it adds to IP is a checksum for the contents of the datagram and port numbers. These are used to give a client/server model – see later.
TCP supplies logic to give a reliable connection-oriented protocol above IP. It provides a virtual circuit that two processes can use to communicate.
In order to use a service, you must be able to find it. The Internet uses an address scheme for machines so that they can be located. The address is a 32 bit integer which gives the IP address.
Class A uses 8 bits for the network address with 24 bits left over for other addressing. Class B uses 16 bit network addressing. Class C uses 24 bit network addressing and class D uses all 32.
Internally, the UNIX network is divided into sub networks. Building 11 is currently on one sub network and uses 10-bit addressing, allowing 1024 different hosts.
8 bits are finally used for host addresses within our subnet. This places a limit of 256 machines that can be on the subnet.
The 32 bit address is usually written as 4 integers separated by dots.
A service exists on a host, and is identified by its port. This is a 16 bit number. To send a message to a server, you send it to the port for that service of the host that it is running on. This is not location transparency! Certain of these ports are “well known”.
A socket is a data structure maintained by the system
to handle network connections. A socket is created using the call socket. It returns an integer that is like a file
descriptor. In fact, under Windows, this handle can be used with Read File and Write File
functions.
#include <sys/types.h>
#include <sys/socket.h>
int socket(int family, int type, int protocol);
Here “family” will be AF_INET for IP communications, protocol will be zero, and type will depend on whether TCP or UDP is used. Two
processes wishing to communicate over a network create a socket each. These are
similar to two ends of a pipe – but the actual pipe does not yet exist.
6.8 JFREE CHART:
JFreeChart is a free 100% Java chart library that makes it easy for developers to display professional quality charts in their applications. JFreeChart’s extensive feature set includes:
A consistent and well-documented API, supporting a wide range of chart types;
A flexible design that is easy to extend, and targets both server-side and client-side applications;
Support for many output types, including Swing components, image files (including PNG and JPEG), and vector graphics file formats (including PDF, EPS and SVG);
JFreeChart is “open source” or, more specifically, free software. It is distributed under the terms of the GNU Lesser General Public Licence (LGPL), which permits use in proprietary applications.
Charts showing values that relate to geographical areas. Some examples include: (a) population density in each state of the United States, (b) income per capita for each country in Europe, (c) life expectancy in each country of the world. The tasks in this project include: Sourcing freely redistributable vector outlines for the countries of the world, states/provinces in particular countries (USA in particular, but also other areas);
Creating an appropriate dataset interface (plus
default implementation), a rendered, and integrating this with the existing
XYPlot class in JFreeChart; Testing, documenting, testing some more,
documenting some more.
Implement a new (to JFreeChart) feature for interactive time series charts — to display a separate control that shows a small version of ALL the time series data, with a sliding “view” rectangle that allows you to select the subset of the time series data to display in the main chart.
There is currently a lot of interest in dashboard displays. Create a flexible dashboard mechanism that supports a subset of JFreeChart chart types (dials, pies, thermometers, bars, and lines/time series) that can be delivered easily via both Java Web Start and an applet.
The property editor mechanism in JFreeChart only
handles a small subset of the properties that can be set for charts. Extend (or
reimplement) this mechanism to provide greater end-user control over the
appearance of the charts.
CHAPTER 7
7.0 APPENDIX
7.1 SAMPLE SCREEN SHOTS:
7.2
SAMPLE SOURCE CODE:
CHAPTER 8
8.1 CONCLUSION AND FUTURE WORK:
In order to detect errors in big data sets from sensor network systems, a novel approach is developed with cloud computing. Firstly error classification for big data sets is presented. Secondly, the correlation between sensor network systems and the scale-free complex networks are introduced. According to each error type and the features from scale-free networks, we have proposed a time-efficient strategy for detecting and locating errors in big data sets on cloud.
Experiment results from our cloud computing environment U-Cloud, it is demonstrated that 1) the proposed scale-free error detecting approach can signifi- cantly reduce the time for fast error detection in numeric big data sets, and 2) the proposed approach achieves similar error selection ratio to non-scale-free error detection approaches. In future, in accordance with error detection for big data sets from sensor network systems on cloud, the issues such as error correction, big data cleaning and recovery will be further explored.
Our experiment results and analysis, it
can be concluded that our proposed error detection approach for big data
processing on cloud can dramatically increase the error detecting speed without
losing error selecting accuracy. Especially, when the error rate for a
targeting big data set is limited and within a small value (1-10 percent ), the
algorithm can efficiently detect the error with high fidelity.