Enabling Fine-grained Multi-keyword Search Supporting Classified Sub-dictionaries over Encrypted Cloud DataAbstract—Using cloud computing, individuals can store their data on remote servers and allow data access to public users through thecloud servers. As the outsourced data are likely to contain sensitive privacy information, they are typically encrypted before uploaded tothe cloud. This, however, significantly limits the usability of outsourced data due to the difficulty of searching over the encrypted data. Inthis paper, we address this issue by developing the fine-grained multi-keyword search schemes over encrypted cloud data. Our originalcontributions are three-fold. First, we introduce the relevance scores and preference factors upon keywords which enable the precisekeyword search and personalized user experience. Second, we develop a practical and very efficient multi-keyword search scheme.The proposed scheme can support complicated logic search the mixed “AND”, “OR” and “NO” operations of keywords. Third, we furtheremploy the classified sub-dictionaries technique to achieve better efficiency on index building, trapdoor generating and query. Lastly,we analyze the security of the proposed schemes in terms of confidentiality of documents, privacy protection of index and trapdoor,and unlinkability of trapdoor. Through extensive experiments using the real-world dataset, we validate the performance of the proposedschemes. Both the security analysis and experimental results demonstrate that the proposed schemes can achieve the same securitylevel comparing to the existing ones and better performance in terms of functionality, query complexity and efficiency.Index Terms—Searchable encryption, Multi-keyword, Fine-grained, Cloud computing.F1 INTRODUCTIONTHE cloud computing treats computing as a utility andleases out the computing and storage capacities to thepublic individuals [1], [2], [3]. In such a framework, theindividual can remotely store her data on the cloud server,namely data outsourcing, and then make the cloud data openfor public access through the cloud server. This represents amore scalable, low-cost and stable way for public data accessbecause of the scalability and high efficiency of cloud servers,and therefore is favorable to small enterprises._ H. Li and Y. Yang are with the School of Computer Science andEngineering, University of Electronic Science and Technology of China,Chengdu, Sichuan, China (e-mail: hongweili@uestc.edu.cn; yangyi.buku@gmail.com)._ H. Li is with State Key Laboratory of Information Security (Institute ofInformation Engineering, Chinese Academy of Sciences, Beijing 100093)(e-mail: hongweili@uestc.edu.cn)._ T. Luan is with the School of Information Technology, Deakin University,Melbourne, Australia(e-mail: tom.luan@deakin.edu.au)._ X. Liang is with the Department of Computer Science, Dartmouth College,Hanover, USA(e-mail: Xiaohui.Liang@dartmouth.edu)._ L. Zhou is with the National Key Laboratory of Science and Technologyon Communication, University of Electronic Science and Technology ofChina, China(e-mail: lzhou@uestc.edu.cn)._ X. Shen is with the Department of Electrical and Computer Engineering,University of Waterloo,Waterloo, Ontario, Canada(e-mail:sshen@uwaterloo.ca).Note that the outsourced data may contain sensitive privacyinformation. It is often necessary to encrypt the private databefore transmitting the data to the cloud servers [4], [5].The data encryption, however, would significantly lower theusability of data due to the difficulty of searching over theencrypted data [6]. Simply encrypting the data may stillcause other security concerns. For instance, Google Searchuses SSL (Secure Sockets Layer) to encrypt the connectionbetween search user and Google server when private data,such as documents and emails, appear in the search results [7].However, if the search user clicks into another website fromthe search results page, that website may be able to identifythe search terms that the user has used.On addressing above issues, the searchable encryption (e.g.,[8], [9], [10]) has been recently developed as a fundamentalapproach to enable searching over encrypted cloud data,which proceeds the following operations. Firstly, the dataowner needs to generate several keywords according to theoutsourced data. These keywords are then encrypted and storedat the cloud server. When a search user needs to access theoutsourced data, it can select some relevant keywords andsend the ciphertext of the selected keywords to the cloudserver. The cloud server then uses the ciphertext to matchthe outsourced encrypted keywords, and lastly returns thematching results to the search user. To achieve the similarsearch efficiency and precision over encrypted data as that ofplaintext keyword search, an extensive body of research hasbeen developed in literature. Wang et al. [11] propose a rankedkeyword search scheme which considers the relevance scores1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing2of keywords. Unfortunately, due to using order-preservingencryption (OPE) [12] to achieve the ranking property, theproposed scheme cannot achieve unlinkability of trapdoor.Later, Sun et al. [13] propose a multi-keyword text searchscheme which considers the relevance scores of keywords andutilizes a multidimensional tree technique to achieve efficientsearch query. Yu et al. [14] propose a multi-keyword top-kretrieval scheme which uses fully homomorphic encryption toencrypt the index/trapdoor and guarantees high security. Caoet al. [6] propose a multi-keyword ranked search (MRSE),which applies coordinate machine as the keyword matchingrule, i.e., return data with the most matching keywords.Although many search functionalities have been developedin previous literature towards precise and efficient searchableencryption, it is still difficult for searchable encryption toachieve the same user experience as that of the plaintextsearch, like Google search. This mainly attributes to followingtwo issues. Firstly, query with user preferences is very popularin the plaintext search [15], [16]. It enables personalized searchand can more accurately represent user’s requirements, but hasnot been thoroughly studied and supported in the encrypteddata domain. Secondly, to further improve the user’s experienceon searching, an important and fundamental function isto enable the multi-keyword search with the comprehensivelogic operations, i.e., the “AND”, “OR” and “NO” operationsof keywords. This is fundamental for search users to prunethe searching space and quickly identify the desired data.Cao et al. [6] propose the coordinate matching search scheme(MRSE) which can be regarded as a searchable encryptionscheme with “OR” operation. Zhang et al. [17] propose aconjunctive keyword search scheme which can be regarded asa searchable encryption scheme with “AND” operation withthe returned documents matching all keywords. However, mostexisting proposals can only enable search with single logicoperation, rather than the mixture of multiple logic operationson keywords, which motivates our work.In this work, we address above two issues by developingtwo Fine-grained Multi-keyword Search (FMS) schemes overencrypted cloud data. Our original contributions can be summarizedin three aspects as follows:• We introduce the relevance scores and the preference factorsof keywords for searchable encryption. The relevancescores of keywords can enable more precise returnedresults, and the preference factors of keywords representthe importance of keywords in the search keyword setspecified by search users and correspondingly enablespersonalized search to cater to specific user preferences. Itthus further improves the search functionalities and userexperience.• We realize the “AND”, “OR” and “NO” operations in themulti-keyword search for searchable encryption. Comparedwith schemes in [6], [13] and [14], the proposedscheme can achieve more comprehensive functionalityand lower query complexity.• We employ the classified sub-dictionaries technique toenhance the efficiency of the above two schemes. Extensiveexperiments demonstrate that the enhanced schemescan achieve better efficiency in terms of index building,trapdoor generating and query in the comparison withschemes in [6], [13] and [14].The remainder of this paper is organized as follows. InSection 2, we outline the system model, threat model, securityrequirements and design goals. In Section 3, we describethe preliminaries of the proposed schemes. We present thedeveloped schemes and enhanced schemes in details in Section4 and Section 5, respectively. Then we carry out the securityanalysis and performance evaluation in Section 6 and Section7, respectively. Section 8 provides a review of the relatedworks and Section 9 concludes the paper.2 SYSTEM MODEL, THREAT MODELAND SECURITY REQUIREMENTS2.1 System ModelAs shown in Fig. 1, we consider a system consists of threeentities.• Data owner: The data owner outsources her data tothe cloud for convenient and reliable data access to thecorresponding search users. To protect the data privacy,the data owner encrypts the original data throughsymmetric encryption. To improve the search efficiency,the data owner generates some keywords for eachoutsourced document. The corresponding index is thencreated according to the keywords and a secret key. Afterthat, the data owner sends the encrypted documents andthe corresponding indexes to the cloud, and sends thesymmetric key and secret key to search users.• Cloud server: The cloud server is an intermediate entitywhich stores the encrypted documents and correspondingindexes that are received from the data owner, andprovides data access and search services to search users.When a search user sends a keyword trapdoor to the cloudserver, it would return a collection of matching documentsbased on certain operations.• Search user: A search user queries the outsourced documentsfrom the cloud server with following three steps.First, the search user receives both the secret key andsymmetric key from the data owner. Second, accordingto the search keywords, the search user uses the secretkey to generate trapdoor and sends it to the cloud server.Last, she receives the matching document collection fromthe cloud server and decrypts them with the symmetrickey.2.2 Threat Model and Security RequirementsIn our threat model, the cloud server is assumed to be “honestbut-curious”, which is the same as most related works onsecure cloud data search [13], [14], [6]. Specifically, the cloudserver honestly follows the designated protocol specification.However, the cloud server could be “curious” to infer andanalyze data (including index) in its storage and messageflows received during the protocol so as to learn additionalinformation. we consider two threat models depending on theinformation available to the cloud server, which are also usedin [13], [6].1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing3Fig. 1. System model• Known Ciphertext Model: The cloud server can onlyknow encrypted document collection C and index collectionI, which are both outsourced from the data owner.• Known Background Model: The cloud server can possessmore knowledge than what can be accessed inthe known ciphertext model, such as the correlationrelationship of trapdoors and the related statistical ofother information, i.e., the cloud server can possess thestatistical information from a known comparable datasetwhich bears the similar nature to the targeting dataset.Similar to [13], [6], we assume search users are trustedentities, and they share the same symmetric key and secretkey. Search users have pre-existing mutual trust with thedata owner. For ease of illustration, we do not considerthe secure distribution of the symmetric key and the secretkey between the data owner and search users; it can beachieved through regular authentication and secure channelestablishment protocols based on the prior security contextshared between search users and the data owner [18]. Inaddition, to make our presentations more focused, we donot consider following issues, including the access controlproblem on managing decryption capabilities given to usersand the data collection’s updating problem on inserting newdocuments, updating existing documents, and deleting existingdocuments, are separated issues. The interested readers onabove issues may refer to [6], [5], [10], [19].Based on the above threat model, we define the securityrequirements as follows:• Confidentiality of documents: The outsourced documentsprovided by the data owner are stored in the cloud server.If they match the search keywords, they are sent to thesearch user. Due to the privacy of documents, they shouldnot be identifiable except by the data owner and theauthorized search users.• Privacy protection of index and trapdoor: As discussed inSection 2.1, the index and the trapdoor are created basedon the documents’ keywords and the search keywords,respectively. If the cloud server identifies the content ofindex or trapdoor, and further deduces any associationbetween keywords and encrypted documents, it may learnthe major subject of a document, even the content of ashort document [20]. Therefore, the content of index andtrapdoor cannot be identified by the cloud server.• Unlinkability of trapdoor: The documents stored in thecloud server may be searched many times. The cloudserver should not be able to learn any keyword informationaccording to the trapdoors, e.g., to determine twotrapdoors which are originated from the same keywords.Otherwise, the cloud server can deduce relationship oftrapdoors, and threaten to the privacy of keywords. Hencethe trapdoor generation function should be randomized,rather than deterministic. Even in case that two searchkeyword sets are the same, the trapdoors should bedifferent.3 PRELIMINARIESIn this section, we define the notation and review the securekNN computation and relevance score, which will serve as thebasis of the proposed schemes.3.1 Notation• F—the document collection to be outsourced, denoted asa set of N documents F = (F1; F2; · · · ; FN).• C—the encrypted document collection according to F,denoted as a set of N documents C = (C1;C2; · · · ;CN).• FID—the identity collection of encrypted documents C,denoted as FID = (FID1; FID2; · · · ; FIDN).• W—the keyword dictionary, including m keywords, denotedas W = (w1;w2; · · · ;wm).• I—the index stored in the cloud server, which is builtfrom the keywords of each document, denoted as I =(I1; I2; · · · ; IN).• fW—the query keyword set generated by a search user,which is a subset of W.• TfW—the trapdoor for keyword set fW.• ]FID—the identity collection of documents returned tothe search user.• FMS(CS)—the abbreviation of FMS and FMSCS.3.2 Secure kNN ComputationWe adopt the work of Wong et al. [21] as our foundation.Wong et al. propose a secure k-nearest neighbor (kNN) schemewhich can confidentially encrypt two vectors and computeEuclidean distance of them. Firstly, the secret key (S;M1;M2)should be generated. The binary vector S is a splitting indicatorto split plaintext vector into two random vectors, whichcan confuse the value of plaintext vector. And M1 and M2 areused to encrypt the split vectors. The correctness and securityof secure kNN computation scheme can be referred to [21].3.3 Relevance ScoreThe relevance score between a keyword and a documentrepresents the frequency that the keyword appears in thedocument. It can be used in searchable encryption for returningranked results. A prevalent metric for evaluating the relevancescore is TF × IDF, where TF (term frequency) representsthe frequency of a given keyword in a document and IDF1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing4(inverse document frequency) represents the importance ofkeyword within the whole document collection. Without lossof generality, we select a widely used expression in [22] toevaluate the relevance score asScore(fW; Fj) =Σw∈fW1|Fj |· (1 + lnfj;w) · ln(1 +Nfw) (1)where fj;w denotes the TF of keyword w in document Fj ;fw denotes the number of documents contain keyword w; Ndenotes the number of documents in the collection; and |Fj |denotes the length of Fj , obtained by counting the number ofindexed keywords.4 PROPOSED SCHEMESIn this section, we firstly propose a variant of the secure kNNcomputation scheme, which serves as the basic framework ofour schemes. Furthermore, we describe two variants of ourbasic framework and the corresponding functionalities of themin detail.4.1 Basic FrameworkThe secure kNN computation scheme uses Euclidean distanceto select k nearest database records. In this section, we presenta variant of the secure kNN computation scheme to achievethe searchable encryption property.4.1.1 InitializationThe data owner randomly generates the secret key K =(S;M1;M2), where S is a (m+1)-dimensional binary vector,M1 and M2 are two (m + 1) × (m + 1) invertible matrices,respectively, and m is the number of keywords in W. Thenthe data owner sends (K; sk) to search users through a securechannel, where sk is the symmetric key used to encryptdocuments outsourced to the cloud server.4.1.2 Index buildingThe data owner firstly utilizes symmetric encryption algorithm(e.g., AES) to encrypt the document collection(F1; F2; · · · ; FN) with the symmetric key sk [23], the encrypteddocument collection are denoted as Cj(j = 1; 2; · · · ;N).Then the data owner generates an m-dimensional binaryvector P according to Cj(j = 1; 2; · · · ;N), where eachbit P[i] indicates whether the encrypted document containsthe keyword wi, i.e., P[i] = 1 indicates yes and P[i] = 0indicates no. Then she extends P to a (m + 1)-dimensionalvector P′, where P′[m + 1] = 1. The data owner usesvector S to split P′ into two (m + 1)-dimensional vectors(pa; pb), where the vector S functions as a splitting indicator.Namely, if S[i] = 0(i = 1; 2; · · · ;m + 1), pa[i] and pb[i]are both set as P′[i]; if S[i] = 1(i = 1; 2; · · · ;m + 1),the value of P′[i] will be randomly split into pa[i] and pb[i](P′[i] = pa[i]+pb[i]). Then, the index of encrypted documentCj can be calculated as Ij = (paM1; pbM2). Finally, the dataowner sends Cj ||FIDj ||Ij (j = 1; 2; · · · ;N) to the cloudserver.4.1.3 Trapdoor generatingThe search user firstly generates the keyword set fW forsearching. Then, she creates a m-dimensional binary vector Qaccording to fW, where Q[i] indicates whether the i-th keywordof dictionary wi is in fW, i.e., Q[i] = 1 indicates yes andQ[i] = 0 indicates no. Further, the search user extends Q toa (m + 1)-dimensional vector Q′, where Q′[m + 1] = −s(the value of −s will be defined in the following schemesin detail). Next, the search user chooses a random numberr > 0 to generate Q′′ = r · Q′. Then she splits Q′′ into two(m + 1) vectors (qa; qb): if S[i] = 0(i = 1; 2; · · · ;m + 1),the value of Q′′[i] will be randomly split into qa[i] and qb[i];if S[i] = 1(i = 1; 2; · · · ;m + 1), qa[i] and qb[i] are bothset as Q′′[i]. Thus, the search trapdoor TfW can be generatedas (M−11 qa;M−12 qb). Then the search user sends TfW to thecloud server.4.1.4 QueryWith the index Ij(j = 1; 2; · · · ;N) and trapdoor TfW, thecloud server calculates the query result asRj = Ij · TfW = (paM1; pbM2) · (M−11 qa;M−12 qb)= pa · qa + pb · qb = P′ · Q′′= rP′ · Q′= r · (P · Q − s)(2)If Rj > 0, the corresponding document identity FIDj willbe returned.Discussions: The Basic Framework has defined the fundamentalsystem structure of the developed schemes. Based onthe secure kNN computation scheme [21], the complementaryrandom parameter r further enhances the security. Differentvalues for parameter s and vectors P and Q can lead to newvariants of the Basic Framework. This will be elaborated inthe follows.4.2 FMS IIn the Basic Framework, P is a m-dimensional binary vector,and each bit P[i] indicates whether the encrypted documentcontains the keyword wi. In the FMS I, the data ownerfirst calculates the relevance score between the keyword wiand document Fj . The relevance score can be calculated asfollows:Score(wi; Fj) =1|Fj |· (1 + lnfj;wi ) · ln(1 +Nfwi) (3)where fj;wi denotes the TF of keyword wi in document Fj ;fwi denotes the number of documents contain keyword wi; Ndenotes the number of documents in the collection; and |Fj |denotes the length of Fj , obtained by counting the number ofindexed keywords.Then the data owner replaces the value of P[i] with thecorresponding relevance score. On the other hand, we alsoconsider the preference factors of keywords. The preferencefactors of keywords indicate the importance of keywords inthe search keyword set personalized defined by the searchuser. For a search user, he may pay more attention to thepreference factors of keywords defined by himself than therelevance scores of the keywords. Thus, our goal is that1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing5if a document has a keyword with larger preference factorthan other documents, it should have a higher priority inthe returned ]FID; and for two documents, if their largestpreference factor keywords are the same, the document withhigher relevance score of the keyword is the better matchingresult.As shown in Fig. 2, we replace the values of P[i] andQ[i] by the relevance score and the preference factor of akeyword, respectively (thus P and Q are no longer binary).The search user can dynamically adjust the preference factorsto achieve a more flexible search. For convenience, the scoreis rounded up, i.e., Score(wi; Fj) = ⌈10 ∗ Score(wi; Fj)⌉,and we assume the relevance score is not more than D,i.e., Score(wi; Fj) < D. For the search keyword set fW =(wn1 ;wn2 ; · · · ;wnl)(1 ≤ n1 < n2 < · · · < nl ≤ m) whichis ordered by ascending importance, the search user randomlychooses a Σ super-increasing sequence (d1 > 0; d2; · · · ; dl) (i.e., j−1i=1 di ·D < dj(j = 2; 3; · · · ; l)), where di is the preferencefactor of keyword wni . Then the search result would be:Rj = r · (P · Q − s) = r · (Σli=1Score(wni ; Fj) · di − s) (4)Theorem 1: (Correctness) For the search keyword set fW =(wn1 ;wn2 ; · · · ;wnl)(1 ≤ n1 < n2 < · · · < nl ≤ m) whichis ordered by ascending preference factors, if F1 contains alarger preference factor keyword compared with F2, then F1has higher priority in the returned ]FID.Proof: For the search keyword set fW =(wn1 ;wn2 ; · · · ;wnl ), assume the keyword sets F1and F2 contain in fW are denoted as fW1 =(wni ; · · · ;wnx )(n1 ≤ ni < · · · < nx ≤ nl) andfW2 = (wnj ; · · · ;wny )(n1 ≤ nj < · · · < ny ≤ nl),respectively, where fW1 and fW2 are both ordered byascending preference factors, and nx > ny. As stated above,Score(wnx Σ ; Fj) ≥ 1 since the score is rounded up, and j−1i=1 di · D < dj(j = 2; 3; · · · ; l). Therefore, there will beR2 = r · (Σwnj∈gW2Score(wnj ; F2) · dj − s)< r · (Σyj=1Score(wnj ; F2) · dj − s)< r · (Σyj=1D · dj − s) < r · (dx − s)< r · (Score(wnx ; F1) · dx − s)< r · (Σwni∈gW1Score(wni ; F1) · di − s)< R1(5)Therefore, F1 has higher priority in the returned ]FID.Theorem 2: (Correctness) For the search keyword set fW =(wn1 ;wn2 ; · · · ;wnl)(1 ≤ n1 < n2 < · · · < nl ≤ m)which is ordered by ascending preference factors, if the largestpreference factor keyword F1 contains is the same as thatF2 contains, and F1 have the higher relevance score of thekeyword, then F1 have higher priority in the returned ]FID.Proof: For the search keyword set fW =(wn1 ;wn2 ; · · · ;wnl ), assume the keyword sets F1 andF2 contain are denoted as fW1 = (wni ; · · · ;wnx )(n1 ≤ni < · · · < nx ≤ nl) and fW2 = (wnj ; · · · ;wnx )(n1 ≤nj < · · · < nx ≤ nl), respectively, where fW1 and fW2are both ordered by ascending preference factors andScore(wnx ; F1) − Score(wnx ; F2) ≥ 1. Thus, there will beR1 =r · (Σwni∈gW1Score(wni ; F1) · di − s)≥ r · (Score(wnx ; F1) · dx − s)(7)R2 =r · (Σwnj∈gW2Score(wnj ; F2) · dj − s) (8)=r · (Score(wnx ; F2) · dx+Σwnj∈gW2−wnxScore(wnj ; F2) · dj − s)<r · (Score(wnx ; F2) · dx +Σwnj∈gW2−wnxD · dj − s)<r · (Score(wnx ; F2) · dx + dx − s)R1 − R2 > r · ((Score(wnx ; F1) − Score(wnx ; F2)) · dx − dx)> r · (dx − dx)> 0 (9)Therefore, F1 have higher priority in the returned ]FID thanF2.Example. We present a concrete example to help understandTheorem 2. The example also illustrates the workingprocess of FMS I. Specifically, we assume that thesearch keyword set is fW = (wn1 ;wn2 ; · · · ;wn5 ), and thelargest preference factor keyword of sets F1 and F2 is thesame, which is wn4 . In addition, we assume the keywordsets F1 and F2 are fW1 = (wn2 ;wn3 ;wn4 ) and fW2 =(wn1 ;wn3 ;wn4 ) respectively. Furthermore, we assume thatthe relevance score is not more than D = 5, and specially,let Score(wn4 ; F1) = 4 and Score(wn4 ; F2) = 2,which satisfy Score(wn4 ; F1) − Score(wn4 ; F2) = 2 ≥1. we randomly choose a super-increasing sequence di ={1; 10; 60; 500; 3000}(i = 1; · · · ; 5), for arbitrary r > 0, therewill beR1 =r · (Σwni∈gW1Score(wni ; F1) · di − s) (11)≥r · (Score(wn4 ; F1) · d4 − s)≥r · (4 · 500 − s)≥r · (2000 − s)1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing6Data owner Cloud server Search user111222 1 2 22keyword dictionary: ( , , )keywords of document : ( , , )corresponding ( , ): ( , , )in : ( , , , , , , , , )( 0 ,0 , , , , , , , ) ji jw wF w wScore w F S Sw w w w wP S S SP Pa ba b= ×××= ××××××××× ××× ××× ×××= ××× ××× ××× ×××® ¢®WWW W1 2( , )( , ) j a ab bp pI = p M p M´ jI_TW__ 2 1 211 22 1 2 2keyword dictionary: ( , , )search keyword set: ( , , )super-increasing sequence: ( , , )in : ( , , , , , , , , )( 0 , 0 , , , , , , l n n nln n nw ww w wd d dw w w w wQ d d da ba b= ×××= ××××××××× ××× ××× ×××= ××× ××× ×××WWW W_ 11 21, )( , )( , ) a ab bQ Q r Q q qT M q M q – -×××® ¢® × ¢®=1 W( )( ( , ) )larger , better result. j ilj ni j iR r P Q sr Score w F d sR== × × -= × _ × -Fig. 2. Structure of the FMS IR2 =r · (Σwnj∈gW2Score(wnj ; F2) · dj − s) (12)=r · (Score(wn4 ; F2) · d4+Σwnj∈gW2−wn4Score(wnj ; F2) · dj − s)<r · (Score(wn4 ; F2) · dx +Σwnj∈gW2−wn4D · dj − s)<r · (Score(wn4 ; F2) · d4 + d4 − s)<r · (2 · 500 + 500 − s)<r · (1500 − s)R1 − R2 >r · (2000 − s) − r · (1500 − s) (13)>r · (2000 − 1500)>500 · r > 04.3 FMS IIIn the FMS II, we do not change the vector P in the BasicFramework, but replace the value of Q[i] by the weight ofsearch keywords, as shown in Fig. 3. With the weight ofkeywords, we can also implement some operations like “OR”,“AND” and “NO” in the Google Search to the searchableencryption.Assume that the keyword sets corresponding to the“OR”, “AND” and “NO” operations are (w′1;w′2; · · · ;w′l1),(w′′1 ;w′′2 ; · · · ;w′′l2) and (w′′′1 ;w′′′2 ; · · · ;w′′′l3), respectively.Denote “OR”, “AND” and “NO” operations by ∨, ∧ and￢, respectively. Thus the matching rule can be representedas (w′1∨ w′2∨ · · · ∨ w′l1) ∧ (w′′1∧ w′′2∧ · · · ∧ w′′l2) ∧(￢w′′′1∧ ￢w′′′2∧ · · · ∧ ￢w′′′l3). For “OR” operation,the search user chooses a super-increasing sequence(a1 > 0; a2; · · · ; al1 )(Σj−1k=1 ak < aj(j = 2; · · · ; l1)) toachieve searching with keyword weight. To enable searchableencryption with “AND” and “NO” operations, the searchuser chooses a sequence (b1; b2; · · · ; bl2 ; c1; c2; · · · ; cl3 ),whereΣl1k=1 Σ ak < bh(h = 1; 2; · · · ; l2) and l1k=1 ak +Σl2h=1 bh < ci(i = 1; 2; · · · ; l3).Assume (w′1;w′2; · · · ;w′l1) are ordered by ascendingimportance, then according to the search keyword set(w′1;w′2; · · · ;w′l1;w′′1 ;w′′2 ; · · · ;w′′l2;w′′′1 ;w′′′2 ; · · · ;w′′′l3),the corresponding values in Q are set as(a1; a2; · · · ; al1 ; b1; b2; · · · ; bl2 ;−c1;−c2; · · · ;−cl3 ). Othervalues in Q are set as 0. Finally, the search user setss =Σl2h=1 bh. In the Query phase, For a document Fj , ifthe corresponding Rj > 0, we claim that Fj can satisfy theabove matching rule.Theorem 3: (Correctness) Fj satisfies the above matchingrule with “OR”, “AND” and “NO” if and only if the correspondingRj > 0.Proof: Firstly, we proof the completeness. Since the weightof w′′′Σ i (i = 1; 2; · · · ; l3) in the vector Q is −ci and ci > l1k=1 ak +Σl2h=1 bh, if any corresponding value of w′′′i in Pof Fj is 1, we can infer P ·Q < 0 and Rj = r·(P ·Q−s) < 0.Therefore, if Rj > 0, any of w′′′i is not in the keyword set ofFj , i.e., Fj satisfies the “NO” operation. Moreover, if Rj > 0,then r · (P · Q − s) = r · (P · Q −Σl2h=1 bh) > 0. Sincebh >Σl1k=1 ak(h = 1; 2; · · · ; l2), all corresponding values ofw′′h in P have to be 1 and at least one corresponding value ofw′k(k = 1; 2; · · · ; l1) in P should be 1. Thus, Fj satisfies the“AND” and “OR” operations. Therefore, if Rj > 0, the vectorP satisfies the operations of “OR”, “AND” and “NO”.Next, we show the soundness. If the vector P satisfiesthe operations of “OR”, “AND” and “NO”, i.e., at least onecorresponding value of keyword w′k in P is 1 (assume thiskeyword is w′(1 ≤  ≤ l1)), all corresponding values ofkeywords w′′h in P are 1 and no corresponding value ofkeyword w′′′i in P is 1. Therefore, Rj = r · (P · Q − s) ≥r · (a + b1 + b2 + · · · + bl2− s) = r · a > 0.Example.We present a concrete example to help understandTheorem 3. The example also illustrates the working processof FMS II. Specifically, we assume that the keyword setscorresponding to the “OR”, “AND” and “NO” operations are(w′1;w′2;w′3), (w′′1 ;w′′2 ;w′′3 ) and (w′′′1 ;w′′′2 ), respectively. Thus,the matching rule can be represented as (w′1∨ w′2∨ w′3) ∧(w′′1∧ w′′2∧ w′′3 ) ∧ (￢w′′′1∧ ￢w′′′2 ). we assume that the searchweights (a1; a2; a3), (b1; b2; b3) and (c1; c2) for “OR”, “AND”and “NO” are(1,5,8), (20,24,96) and (-500,-600), respectively.We firstly prove Rj > 0 when Fj satisfies the matchingrule. Specifically, assume that Fj satisfies the matching rulew′2∧(w′′1∧w′′2∧w′′3 )∧(￢w′′′1∧￢w′′′2 ). Thus the correspondingvalues of vector P are (0; 1; 0), (1; 1; 1) and (0; 0), respectively.Thus, the result of s =Σ3h=1 bh = 20 + 24 + 96 = 140,1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing7Data owner Cloud server Search user´ jI_T1 2 2 1 12 2 W1 2keyword dictionary: ( , , )keywords of document : ( , , )in : ( , , , , , , , , )( 0, 0 , , 1 , , 1 , , 1 , )( , )( , ) jj a ab bw wF w ww w w w wPP P p pI p M p Ma b= ×××= ×××××× ××× ××× ×××= ××× ××× ××× ×××® ¢®=WWW W 12 121 2 1122 11 22 1 2keyword dictionary: ( , , )operation keyword set keyword weightOR ( , , , ) ( , , , )AND ( , , , ) ( , , , )NO ( , , , ll llw ww w w a a aw w w b b bw w= ×××¢ ¢ ××× ¢ ×××¢ ¢ ××× ¢ ×××¢¢ ¢¢ ×××W_1 2 3 1 2 311 21) ( , , , )in : ( , , , , , , , , )( 0, 0 , , , , , , , )( , )( , ) l la ab bw c c cw w w w wQ a c bQ Q r Q q qT M q M qa b ca b c- -¢¢ ×××××× ¢ ××× ¢¢ ××× ¢ ×××= ××× ××× – ××× ×××® ¢® × ¢®=WW( )( )0 satisfy “OR”,”AND” and “NO”larger larger weight of “OR” j j jR r P Q sr a c b sRRa b c= × × -= × ×××+ +×××- +×××+ + ×××-> __Fig. 3. Structure of the FMS IIfor arbitrary r > 0, the result of Rj will beRj = r · (P · Q − s)= r · (a2 + b1 + b2 + b3 − s)= r · (5 + 20 + 24 + 96 − 140)= 5r > 0(14)From the above example, we can easily see that Rj > 0when Fj satisfies the matching rule. Next, we show thatRj < 0 when Fj does not satisfy the matching rule. Especially,we assume that the ”AND” operation does not satisfy thematching rule. Here, we set the first keyword does not matchthe rule, therefore the search keyword set of ”AND” operationsare (0; 1; 1) instead of (1; 1; 1). Thus, the result of Ri will beRj = r · (P · Q − s)= r · (a2 + b2 + b3 − s)= r · (5 + 24 + 96 − 140)= −15r < 0(15)Obviously, Rj < 0 when Fj does not satisfy the matchingrule.5 ENHANCED SCHEMEIn practice, apart from some common keywords, other keywordsin dictionary are generally professional terms, and thispart of the dictionary will rapidly increase when the dictionarybecomes larger and more comprehensive. Simultaneously, thedata owner’s index will become longer, although many dimensionsof keywords will never appear in her documents.That will cause redundant computation and communicationoverhead.In this section, we further propose a Fine-grained MultikeywordSearch scheme supporting Classified Sub-dictionaries(FMSCS), which classifies the total dictionary as a commonsub-dictionary and many professional sub-dictionaries. Ourgoal is to significantly reduce the computation and communicationoverhead. We have researched in a file set randomlychosen from the National Science Foundation (NSF) ResearchAwards Abstracts 1990-2003 [24]. As shown in Fig. 4, weclassify the total dictionary to many sub-dictionaries suchas common sub-dictionary, computer science sub-dictionary,mathematics sub-dictionary and physics sub-dictionary, etc.And the search process will only be some minor changes inInitialization.Change of Initialization: Compared with theBasic Framework, in the enhanced scheme thedata owner should firstly choose corresponding subdictionaries.Then her own dictionary can be combined as{f1||Subdic1||f2||Subdic2|| · · · }, where Subdici representsall keywords contained in corresponding sub-dictionary andfi is filling factor with random length which will be 0 stringin the index, the filling factor is used to confuse length ofthe data owner’s own dictionary and relative positions of subdictionaries.Then, the data owner and search user will use thisdictionary to generate the index and trapdoor, respectively.Note that in an dictionary, two professional sub-dictionariescan even contain a same keyword, but only the first appearedkeyword will be used to generate index and trapdoor, anotherwill be set to 0 in the vector. And the secret key K willbe formed as (S;M1;M2; |f1|;DID1 ; |f2|;DID2 ; · · · ), whereDIDi represents the identity of sub-dictionary and |fi| is thelength of fi. Other than these changes, the remaining phases(i.e., Index building, Trapdoor generating and Query) aresame as the Basic Framework.Dictionary Updating: In the searchable encryptionschemes with dictionary, dictionary update is a challengeproblem because it may cause to update massive indexesoutsourced to the cloud server. In general dictionary-basedsearch schemes, e.g., [13] and [14], the update of dictionarywill lead to re-generation of all indexes. In our FMSCSschemes, when it needs to change the sub-dictionaries or addnew sub-dictionaries, only the data owners who use the correspondingsub-dictionaries need to update their indexes, mostother data owners do not need to do any update operations.Such dictionary update operations are particularly lightweight.In addition, Li et al. [9] utilize the dimension expansiontechnique to implement the efficient dictionary expansion.Such method can also be included into our dictionary updatingprocess. And our scheme can even be more efficient than [9]since although [9] does not need to re-generate all indexes,but the corresponding extended operations on all indexes arenecessary. In comparison, our schemes only need to extendthe indexes of partial data owners.1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing8________ ______    _________ ___________ _________ _______     ___ _______ _        ____ _________ ______________ _     _______  ___ ________ __                _              __ ______ _______ ___         __ _______ _______________ ___       __ __________ __________ ______ ___ ___ ___ _________ _!          _” __________1 2 3 4Dictionary: || common || || computer science f f || f ||mathematics || f 1 2 3 4Dictionary: f ¢|| physics || f ¢ || mathematics || f ¢|| common || f ¢ _________ __________            __Fig. 4. Classified sub-dictionaries6 SECURITY ANALYSISIn this section, we analyze the main security properties ofthe proposed schemes. In particular, our analysis focuseson how the proposed schemes can achieve confidentialityof documents, privacy protection of index and trapdoor, andunlinkability of trapdoor. Other security features are not thefocus of our concern.6.1 Confidentiality of DocumentsIn our schemes, the outsourced documents are encrypted bythe traditional symmetric encryption algorithm (e.g., AES). Inaddition, the secret key sk is generated by the data owner andsent to the search user through a secure channel. Since theAES encryption algorithm is secure [23], any entity cannotrecover the encrypted documents without the secret key sk.Therefore, the confidentiality of encrypted documents can beachieved.6.2 Privacy Protection of Index and TrapdoorAs shown in Section 4.1, both the index Ij = (paM1; pbM2)and the trapdoor TfW = (M−11 qa;M−12 qb) are ciphertextsof vectors (P;Q). The secret key is K = (S;M1;M2) inthe FMS or (S;M1;M2; |f1|;DID1 ; |f2|;DID2 ; · · · ) in theFMSCS, where S functions as a splitting indicator whichsplits P and Q into (pa; pb) and (qa; qb), respectively, twoinvertible matrices M1 and M2 are used to encrypt (pa; pb)and (qa; qb). The security of this encryption algorithm has beenproved in the known ciphertext model [21]. Thus, the contentof index and trapdoor cannot be identified. Therefore, privacyprotection of index and trapdoor can be achieved.6.3 Unlinkability of TrapdoorTo protect the security of search, the unlinkability of trapdoorshould be achieved. Although the cloud server cannotdirectly recover the keywords, the linkability of trapdoor maycause leakage of privacy, e.g., the same keyword set may besearched many times, if the trapdoor generation function isdeterministic, even though the cloud server cannot decryptthe trapdoors, it can deduce the relationship of keywords. Weconsider whether the trapdoor TfW = (M−11 qa;M−12 qb) can belinked to the keywords. We prove our schemes can achieve theunlinkability of trapdoor in a strong threat model, i.e., knownbackground model [6].Known Background Model: In this model, the cloudserver can possess the statistical information from a knowncomparable dataset which bears the similar nature to thetargeting dataset.TABLE 1Structure of Q′Q′[1] _ _ _Q′[m] Q′[m + 1]FMS(CS) I _ _ _ 0 _ _ _ di _ _ _ 0 _ _ _ dj _ _ _ 􀀀sFMS(CS) II _ _ _ ak _ _ _ bh _ _ _ 0 _ _ _ ci _ _ _ 􀀀sAs shown in Table 1, in our FMS(CS) I, the trapdooris constituted by two parts. The values of all dimensionsdi(i = 1; 2; · · · ; l) are the super-increasing sequence randomlychosen by the search user (assume there are _ possiblesequences). And the (m+ 1) dimension is −s defined by thesearch user, where s is a positive random number. Assumethe size of −s is _s bits, there are 2_s possible values for−s. Further, to generate Q′′ = r · Q′, Q′ is multiplied by apositive random number r, there are 2_r possible values forr (if the search user chooses _r-bit r). Finally, Q′′ is splitto (qa; qb) according the splitting indicator S. Specifically, ifS[i] = 0(i = 1; 2; · · · ;m + 1), the value of Q′′[i] will berandomly split into qa[i] and qb[i], assume in S the numberof ‘0’ is _, and each dimension of qa and qb is _q bits. Notethat _s, _r, _ and _q are independent of each other. Thenin our FMS(CS) I, we can compute the probability that twotrapdoors are the same as follows:P1 =1_ · 2_s · 2_r · (2_q )_ =1_ · 2_s+_r+__q(16)Therefore, the larger _, _s, _r, _ and _q can achieve thestronger security, e.g., if we choose 1024-bit r, then the1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing9probability P1 < 1=21024. As a result, the probability thattwo trapdoors are the same is negligible.And in the FMS(CS) II, because −s = −Σl2h=1 bh,its value depends on the weight sequence(a1; a2; · · · ; al1 ; b1; b2; · · · ; bl2 ; c1; c2; · · · ; cl3 ). Assumethe number of different sequences is denoted as _, then wecan compute:P2 =1_ · 2_r · (2_q )_ =1_ · 2_r+__q(17)Similarly, in the FMS(CS) II and the FMS(CS) III, the probabilitythat two trapdoors are the same is negligible. Therefore,in our schemes, the unlinkability of trapdoor can be achieved.In summary, we present the comparison results of securitylevel in Table 2, where I and II represent FMS(CS) I andFMS(CS) II, respectively. It can be seen that all schemes canachieve confidentiality of documents and privacy protectionof index and trapdoor, but the OPE schemes [11], [25] cannotachieve the unlinkability of trapdoor very well because of thesimilarity relevance mentioned in [14].TABLE 2Comparison of Security Level[11], [25] [6], [13], [14] I IIConfidentialityp p p pPrivacy protectionp p p pUnlinkabilityp p pDiscussions: In MRSE [6], the values of P ·Q are equal tothe number of matching keywords, which suffers scale analysisattack when the cloud server is powerful and has knowledgeof some background information. To solve this problem, itextends the index and inserts a random number “j whichfollows a normal distribution and can confuse the values ofP ·Q. Thus, enhanced MRSE can resist scale analysis attack.However, the introduction of “j causes precision decrease ofthe returned results. There is a trade-off between precisionand security in MRSE. In comparison, our schemes do notsuffer the scale analysis attack. Because the values of P · Qin our schemes do not disclose any information due to therandomly selected sequences mentioned in Section 4.2 andSection 4.3. Therefore, our proposal can achieve the securitywithout sacrificing precision.7 PERFORMANCE EVALUATIONIn this section, we evaluate the performance of the proposedschemes using simulations, and compare the performance withthat of existing proposals in [6], [13], [14]. We apply a realworlddataset from the National Science Foundation ResearchAwards Abstracts 1990-2003 [24], in which we random selectmultiple documents and conduct real-world experiments on anIntel Core i5 3.2 GHz system.7.1 FunctionalityWe compare functionalities between [6], [13], [14] and ourschemes in Table 3, where I and II represent FMS(CS) I andFMS(CS) II, respectively.MRSE [6] can achieve multi-keyword search and coordinatematching using secure kNN computation scheme. And [13]and [14] considers the relevance scores of keywords. Comparedwith the other schemes, our FMS(CS) I considers boththe relevance scores and the preference factors of keywords.Note that if the search user sets all relevance scores andpreference factors of keywords as the same, the FMS(CS) Idegrades to MRSE and the coordinate matching can beachieved. And in the FMS(CS) II, if the search user sets allpreference factors of “OR” operation keywords as the same,the FMS(CS) II can also achieve the coordinate matchingof “OR” operation keywords. Particularly, the FMS(CS) IIachieves some fine-grained operations of keyword search,i.e., “AND”, “OR” and “NO” operations in Google Search,which are definitely practical and significantly enhance thefunctionalities of encrypted keyword search.TABLE 3Comparison of Functionalities[6] [13] [14] I IIMulti-keyword searchp p p p pCoordinate matchingp p p p pRelevance scorep p pPreference factorp pAND OR NO operationsp7.2 Query ComplexityIn the FMS(CS) II, we can implement “OR”, “AND” and“NO” operations by defining appropriate weights of keywords,this scheme provides a more fine-grained search than [6],[13] and [14]. If the keywords to perform “OR”, “AND” and“NO” operations are (w′1;w′2; · · · ;w′l1), (w′′1 ;w′′2 ; · · · ;w′′l2)and (w′′′1 ;w′′′2 ; · · · ;w′′′l3), respectively. Our FMS(CS) II cancomplete the search with only one query, however, in [6],[13] and [14], they would complete the search through thefollowing steps:• For the “OR” operation of l1 keywords, they need onlyone query Query(w′1;w′2; · · · ;w′l1) to return a collectionof documents with the most matching keywords (i.e.,coordinate matching), which can be denoted as X =Query(w′1;w′2; · · · ;w′l1).• For the “AND” operation of l2 keywords, [6], [13]and [14] cannot generate a query for multiple keywordsto achieve the “AND” operation. Therefore, aftercosting l2 queries Query(w′′i )(i = 1; 2; · · · ; l2),they can do the “AND” operation, and the correspondingdocument set can be denoted as Y =Query(w′′1 )∩Query(w′′2 )∩· · ·∩Query(w′′l2).• For the “NO” operation of l3 keywords, they need l3queries Query(w′′′i )(i = 1; 2; · · · ; l3), firstly. Then, thedocument set of the “NO” operation can be denoted asZ = Query(w′′′1 )∩Query(w′′′2 )∩· · ·∩Query(w′′′l3).• Finally, the document collection achieved “OR”, “AND”and “NO” operations can be represented as X∩Y∩Z.As shown in Fig. 5a, 5b and 5c, to achieve these operations,the FMS(CS) II can outperform the existing proposals withless queries generated.1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing102468102468100510152025Number Number of “NO” keywords of “AND” keywordsNumber of queries[6] & [13] & [14]FMS(CS)_II(a)24681024681005101520Number of “NO” keywords Number of “OR” keywordsNumber of queries[6] & [13] & [14]FMS(CS)_II(b)24681024681005101520Number of “OR” keywords Number of “AND” keywordsNumber of queries[6] & [13] & [14]FMS(CS)_II(c)Fig. 5. Time for building index. (a) Number of queriesfor the different number of “AND” and “NO” keywordswith the same number of “OR” keywords, l1 = 5. (b)Number of queries for the different number of “OR” and“NO” keywords with the same number of “AND” keywords,l2 = 5. (c) Number of queries for the different number of“AND” and “OR” keywords with the same number of “NO”keywords, l3 = 5.7.3 Efficiency7.3.1 Computation overheadIn order to easily demonstrate our scheme computation overhead,we analysis our scheme from each phase.Index building. Note that the Index building phase of [6]is the same as our FMS II scheme, without calculating therelevance score. And the Index building phase of the FMS Iis the same as [13], containing the relevance score computing.Compared with the FMS I, the FMS II does not need to calculatethe relevance score. And compared with the computationcost of building index, the cost of calculating the relevancescore is negligible, we do not distinguish them. Moreover,in our enhanced schemes (FMSCS), we divide the totaldictionary into 1 common sub-dictionary and 20 professionalsub-dictionaries (assume each data owner averagely chooses 1common sub-dictionary and 3 professional sub-dictionaries togenerate the index). As shown in Fig. 6, we can see the time forbuilding index is dominated by both the size of dictionary andthe number of documents. And compared with [6], [13], [14]and our FMS schemes, the FMSCS schemes largely reducethe computation overhead.Trapdoor generating. In Trapdoor generating phase, [6]and [13] firstly creates a vector according to the searchkeyword set fW, then encrypts the vector by the secure kNNcomputation scheme. And [14] also generates a vector anduses homomorphic encryption to encrypt each dimension. Incomparison, our FMS I and FMS II schemes should firstly2000 4000 6000 8000 100000500100015002000Size of dictionaryTime (s)[6] & [13] & FMS[14]FMSCS(a)2000 4000 6000 8000 100000100200300400500600Number of documentsTime (s)[6] & [13] & FMS[14]FMSCS(b)Fig. 6. Time for building index. (a) For the different size ofdictionary with the same number of documents, N=6000.(b) For the different number of documents with the samesize of dictionary, |W| = 4000.2000 4000 6000 8000 10000020040060080010001200Size of dictionaryTime (ms)[6] & [13] & FMS[14]FMSCS(a)10 20 30 40 50050100150200250300Number of query keywordsTime (ms)[6] & [13] & FMS[14]FMSCS(b)Fig. 7. Time for generating trapdoor. (a) For the differentsize of dictionary with the same number of query keywords,|fW|=20. (b) For the different number of query keywordswith the same size of dictionary, |W| = 4000.generate a super-increasing sequence and a weight sequence,respectively. But actually, we can pre-select a correspondingsequence for each scheme, it can also achieve search processand privacy. Because even if the vectors are the same formultiple queries, the trapdoors will be not the same due tothe security of kNN computation scheme. Therefore, the computationcost of [6], [13] and all FMS schemes in Trapdoorgenerating phase are the same. As shown in Fig. 7, the timefor generating trapdoor is dominated by the size of dictionary,instead of the number of query keywords. Hence, our FMSCSschemes are also very efficient in Trapdoor generating phase.Query. As [6], [13] and the FMS all adopt the secure kNNcomputation scheme, the time for query is the same. Thecomputation overhead in Query phase, as shown in Fig. 8,is greatly affected by the size of dictionary and the numberof documents, and almost has no relation to the number ofquery keywords. Further we can see, our FMSCS schemessignificantly reduce the computation cost in Query phase.As [14] needs to encrypt each dimension of index/trapdoorusing full homomorphic encryption, its index/trapdoor size isenormous. Note that, in Trapdoor generating and Queryphases, the computation overheads are not affected by thenumber of query keywords. Thus our FMS and FMSCSschemes are more efficient compared with some multiplekeywordsearch schemes [26], [27], as their cost is linear withthe number of query keywords.1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing112000 4000 6000 8000 10000010203040506070Size of dictionaryTime (s)[6] & [13] & FMS[14]FMSSCS(a)2000 4000 6000 8000 1000001020304050Number of documentsTime (s)[6] & [13] & FMS[14]FMSCS(b)10 20 30 40 5005101520253035Number of query keywordsTime (s)[6] & [13] & FMS[14]FMSCS(c)Fig. 8. Time for query. (a) For the different size ofdictionary with the same number of documents and numberof search keywords, N = 6000; |fW| = 20. (b) Forthe different number of documents with the same sizeof dictionary and number of search keywords, |W| =4000; |fW| = 20. (c) For the different number of searchkeyword with the same size of dictionary and number ofdocuments, N = 6000; |W| = 4000.7.3.2 Storage overheadAs shown in Table 4, we provide a comparison of storageoverhead among several schemes. Specifically, we evaluate thestorage overhead from three parts: the data owner, the searchuser and the cloud server.According to Table 4, in the FMS, the FMSCS as well asschemes of [6] and [13], the storage overhead of the dataowner are the same. In these schemes, the data owner preservesher secret key K = (S;M1;M2) and symmetric key sklocally, where S is an (m+1)-dimensional vector, M1 and M2are (m+1)×(m+1) invertible matrices. All elements in S,M1and M2 are the float number. Since the size of a float numberis 4 bytes, the size of K is 4· (m+1)+8· (m + 1)2 bytes. Weassume that the size of sk is Ssk that is a constant. Thus, thetotal size of storage overhead is 4·(m+1)+8·(m + 1)2+Sskbytes. However, in [14], the storage overhead of data owneris _5=8 bytes, where the _ is the secure parameter. Thestorage overhead is 4GB when we choose _ = 128, which ispopular in a full homomorphic encryption scheme. However,the storage overhead of the FMS and the FMSCS are almost763MB when we choose m = 10000, which is large enoughfor a search scheme. Therefore, the FMS and the FMSCS aremore efficient than scheme in [14] in terms of the storageoverhead of the data owner.As shown in Table 4, a search user in the FMS, the FMSCSas well as the schemes of [6] and [13] preserves the secret keyK = (S;M1;M2) and the symmetric key sk locally. Therefore,the total storage overhead is 4(m+1)+8(m + 1)2+Sskbytes. However, in [14], the storage overhead is _5=8 + _2=8bytes. The storage overhead is 4GB when we choose _ = 128,which is popular in a full homomorphic encryption scheme.However, the storage overhead of the FMS and the FMSCSare almost 763MB when we choose m = 10000, which islarge enough for a search scheme. Therefore, the FMS andthe FMSCS are more efficient than scheme in [14] in termsof the storage overhead of the search user.The cloud server preserves the encrypted documents and theindexes. The size of encrypted documents in all schemes arethe same, i.e., N·Ds. For the indexes, in the FMS and schemesin [6] and [13], the storage overhead are 8 · (m+1) ·N bytes.In the FMSCS, the storage overhead is 8·“· (m+1) ·N bytes,where 0 < ” < 1. When m = 1000 and N = 10000 whichare large enough for a search scheme, the storage overhead ofindexes is about 132MB in the FMSCS. And in schemes of [6]and [13] as well as the FMS, the size of indexes is 760MB withthe same conditions. In scheme in [14], the storage overheadof indexes is N · Ds + m · N · (_=8)5 bytes, it is 4GB whenwe choose _ = 128, which is popular in a full homomorphicencryption scheme. Therefore, the FMS and the FMSCS aremore efficient than scheme in [14] in terms of the storageoverhead of the cloud server.7.3.3 Communication overheadAs shown in Table 5, we provide a comparison of communicationoverhead among several schemes. Specifically,we consider the communication overhead from three parts:the communication between the data owner and the cloudserver (abbreviated as D-C), the communication between thesearch user and the cloud server (abbreviated as C-S) and thecommunication between the data owner and the search user(abbreviated as D-S).D-C. In the FMS as well as schemes of [6] and [13], the dataowner needs to send information to cloud server in the formof Cj ||FIDj ||Ij (j = 1; 2; · · · ;N), where the Cj representsthe encrypted documents, FIDj represents the identity of thedocument and Ij represents the index. We assume that theaverage size of documents is Ds, thus the size of documentsis N ·Ds. We assume the encrypted documents identity FIDis a 10-byte string. Thus, the total size of the identity FIDis 10 · N bytes. The index Ij = (paM1; pbM2) contains two(m+1)-dimensional vectors. Each dimension is a float number(the size of each float is 4 bytes). Thus, the total size of index is8·(m+1)·N bytes. Therefore, the total size of communicationoverhead is 8·(m+1)·N+10·N+N·Ds bytes. In the FMSCS,the total size of communication overhead is 8·“· (m+1)·N +10·N+N·Ds bytes. If we choose the ” as 0:2, the size of indexis 1:6 · (m+1) ·N bytes, and the total size of communicationof FMSCS is 1:6·(m+1)·N+10·N+Ds ·N bytes. However,in [14], the communication overhead is N ·Ds +m·N · _5=8bytes, where _ is the secure parameter. If we choose _ = 128which is popular in a full homomorphic encryption schemeand m = 1000 and N = 10000 which are large enough fora search scheme, the FMS and the FMSCS are more efficientthan scheme in [14] in terms of the communication overheadof D-C.1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing12TABLE 4Comparison of Storage Overhead (Bytes). (m represents the size of dictionary; N represents the number ofdocuments; Ds represents the average size of each encrypted document; _ represents the secure parameter; “represents the decrease rate of dictionary by using our classified sub-dictionaries technology; Ssk represents the sizeof symmetric key.)[14] [6], [13] and FMS FMSCSData Owner _5=8 4 _ (m + 1) + 8 _ (m + 1)2 + Ssk 4 _ (m + 1) + 8 _ (m + 1)2 + SskSearch User _5=8 + _2=8 4 _ (m + 1) + 8 _ (m + 1)2 + Ssk 4 _ (m + 1) + 8 _ (m + 1)2 + SskCloud Server N _ Ds + m _ N _ _5=8 N _ Ds + 8 _ (m + 1) _ N N _ Ds + 8 _ ” _ (m + 1) _ NTABLE 5Comparison of Communication Overhead (Bytes). (m represents the size of dictionary; N represents the number ofdocuments; Ds represents the average size of each encrypted document; T represents the number of returneddocuments; _ represents the secure parameter; ” represents the decrease rate of dictionary by using our classifiedsub-dictionaries technology; Ssk represents the size of symmetric key.)[14] [6], [13] and FMS FMSCSD-C N _ Ds + m _ N _ _5=8 8 _ (m + 1) _ N + 10 _ N + N _ Ds 8 _ ” _ (m + 1) _ N + 10 _ N + N _ DsC-S m _ _5=8 + T _ Ds 8 _ (m + 1) + T _ Ds 8 _ ” _ (m + 1) + T _ DsD-S _5=8 + _2=8 4 _ (m + 1) + 8 _ (m + 1)2 + Ssk 4 _ (m + 1) + 8 _ (m + 1)2 + SskC-S. The C-S consists of two phases: Query and Resultsreturning. In the Query phase, a search user in the FMS as wellas the schemes in [6] and [13] sends the trapdoor to the cloudserver in the form of TfW = (M−11 qa;M−12 qb), which containstwo (m+1)-dimensional vectors. Thus, the communicationoverhead is 8·(m+1) bytes. In the FMSCS, the communicationoverhead is 8 · ” · (m + 1)(0 < ” < 1) bytes. In the Resultsreturning phase, the cloud server sends the correspondingresult to the search user. The communication overhead of CSincreases along with the number of returned documentsat this point. We assume that the number of the returneddocuments is T, thus, the total communication overhead ofcloud server to search user is T · Ds bytes. Therefore, thetotal communication overhead of C-S is 8 ·m+T ·Ds bytes.In the FMS as well as the schemes in [6] and [13], the totalcommunication overhead of C-S is 8 · ” · (m + 1) + T · Dsbytes. In [14], the total communication overhead of C-S ism·_5=8+T ·Ds bytes. If we choose _ = 128 which is popularin a full homomorphic encryption scheme and m = 1000 andN = 10000 which are large enough for a search scheme, theFMS and the FMSCS are more efficient than scheme in [14]in terms of the communication overhead of C-S.D-S. From table 5, we can see that the communicationoverhead of the FMS, the FMSCS as well as schemes in[6] and [13] are the same. In the Initialization phase, thedata owner sends the secret key K = (S;M1;M2) andsymmetric key sk to the search user, where S is an (m+ 1)-dimensional vector, M1 and M2 are (m + 1) × (m + 1)invertible matrices. Thus, the size of the secret key K is4 · (m + 1) + 8 · (m + 1)2 bytes. Therefore, the total sizeof communication overhead is 4 · (m+1)+8 · (m + 1)2+Sskbytes, where the Ssk represents the size of symmetric key.However, the communication overhead of scheme in [14] is_5=8+_2=8 bytes. The communication overhead is 4GB whenwe choose _ = 128, which is popular in a full homomorphicencryption scheme. However, the communication overhead ofthe FMS and the FMSCS are almost 763MB when we choosem = 10000, which is large enough for a search scheme.Therefore, the FMS and the FMSCS are more efficient thanscheme in [14] in terms of the communication overhead ofD-S.8 RELATED WORKThere are mainly two types of searchable encryption in literature,Searchable Public-key Encryption (SPE) and SearchableSymmetric Encryption (SSE).8.1 SPESPE is first proposed by Boneh et al. [28], which supportssingle keyword search on encrypted data but the computationoverhead is heavy. In the framework of SPE, Boneh et al. [27]propose conjunctive, subset, and range queries on encrypteddata. Hwang et al. [29] propose a conjunctive keyword schemewhich supports multi-keyword search. Zhang et al. [17] proposean efficient public key encryption with conjunctivesubsetkeywords search. However, these conjunctive keywordsschemes can only return the results which match all thekeywords simultaneously, and cannot rank the returned results.Qin et al. [30] propose a ranked query scheme which usesa mask matrix to achieve cost-effectiveness. Yu et al. [14]propose a multi-keyword top-k retrieval scheme with fullyhomomorphic encryption, which can return ranked results andachieve high security. In general, although SPE allows moreexpressive queries than SSE [13], it is less efficient, andtherefore we adopt SPE in the work.8.2 SSEThe concept of SSE is first developed by Song et al. [8].Wang et al. [25] develop the ranked keyword search scheme,which considers the relevance score of a keyword. However,the above schemes cannot efficiently support multi-keywordsearch which is widely used to provide the better experience1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing13to the search user. Later, Sun et al. [13] propose a multikeywordsearch scheme which considers the relevance scoresof keywords, and it can achieve efficient query by utilizingthe multidimensional tree technique. A widely adopted multikeywordsearch approach is multi-keyword ranked search(MRSE) [6]. This approach can return the ranked results ofsearching according to the number of matching keywords. Liet al. [10] utilize the relevance score and k-nearest neighbortechniques to develop an efficient multi-keyword searchscheme that can return the ranked search results based on theaccuracy. Within this framework, they leverage an efficientindex to further improve the search efficiency, and adopt theblind storage system to conceal access pattern of the searchuser. Li et al. [19] also propose an authorized and ranked multikeywordsearch scheme (ARMS) over encrypted cloud databy leveraging the ciphertext policy attribute-based encryption(CP-ABE) and SSE techniques. Security analysis demonstratesthat the proposed ARMS scheme can achieve collusion resistance.In this paper, we propose FMS(CS) schemes which notonly support multi-keyword search over encrypted data, butalso achieve the fine-grained keyword search with the functionto investigate the relevance scores and the preference factors ofkeywords and, more importantly, the logical rule of keywords.In addition, with the classified sub-dictionaries, our proposalis efficient in terms of index building, trapdoor generating andquery.9 CONCLUSIONIn this paper, we have investigated on the fine-grained multikeywordsearch (FMS) issue over encrypted cloud data, andproposed two FMS schemes. The FMS I includes both therelevance scores and the preference factors of keywords toenhance more precise search and better users’ experience,respectively. The FMS II achieves secure and efficient searchwith practical functionality, i.e., “AND”, “OR” and “NO”operations of keywords. Furthermore, we have proposed theenhanced schemes supporting classified sub-dictionaries (FMSCS)to improve efficiency.For the future work, we intend to further extend the proposalto consider the extensibility of the file set and the multi-usercloud environments. Towards this direction, we have madesome preliminary results on the extensibility [5] and the multiusercloud environments [19]. Another interesting topic is todevelop the highly scalable searchable encryption to enableefficient search on large practical databases.ACKNOWLEDGMENTThis work is supported by the National Natural ScienceFoundation of China under Grants 61472065, 61350110238,61103207, U1233108, U1333127, and 61272525, the InternationalScience and Technology Cooperation and ExchangeProgram of Sichuan Province, China under Grant 2014HH0029,China Postdoctoral Science Foundation funded projectunder Grant 2014M552336, and State Key Laboratory ofInformation Security Open Foundation under Grant 2015-MS-02.REFERENCES[1] H. Liang, L. X. Cai, D. Huang, X. Shen, and D. Peng, “An smdpbasedservice model for interdomain resource allocation in mobile cloudnetworks,” IEEE Transactions on Vehicular Technology, vol. 61, no. 5,pp. 2222–2232, 2012.[2] M. M. Mahmoud and X. Shen, “A cloud-based scheme for protectingsource-location privacy against hotspot-locating attack in wireless sensornetworks,” IEEE Transactions on Parallel and Distributed Systems,vol. 23, no. 10, pp. 1805–1818, 2012.[3] Q. Shen, X. Liang, X. Shen, X. Lin, and H. Luo, “Exploiting geodistributedclouds for e-health monitoring system with minimum servicedelay and privacy preservation,” IEEE Journal of Biomedical and HealthInformatics, vol. 18, no. 2, pp. 430–439, 2014.[4] T. Jung, X. Mao, X. Li, S.-J. Tang, W. Gong, and L. Zhang, “Privacypreservingdata aggregation without secure channel: multivariate polynomialevaluation,” in Proceedings of INFOCOM. IEEE, 2013, pp.2634–2642.[5] Y. Yang, H. Li, W. Liu, H. Yang, and M. Wen, “Secure dynamicsearchable symmetric encryption with constant document update cost,”in Proceedings of GLOBCOM. IEEE, 2014, to appear.[6] N. Cao, C. Wang, M. Li, K. Ren, and W. Lou, “Privacy-preserving multikeywordranked search over encrypted cloud data,” IEEE Transactionson Parallel and Distributed Systems, vol. 25, no. 1, pp. 222–233, 2014.[7] https://support.google.com/websearch/answer/173733?hl=en.[8] D. X. Song, D. Wagner, and A. Perrig, “Practical techniques for searcheson encrypted data,” in Proceedings of S&P. IEEE, 2000, pp. 44–55.[9] R. Li, Z. Xu, W. Kang, K. C. Yow, and C.-Z. Xu, “Efficient multikeywordranked query over encrypted data in cloud computing,” FutureGeneration Computer Systems, vol. 30, pp. 179–190, 2014.[10] H. Li, D. Liu, Y. Dai, T. H. Luan, and X. Shen, “Enabling efficientmulti-keyword ranked search over encrypted cloud data through blindstorage,” IEEE Transactions on Emerging Topics in Computing, 2014,DOI10.1109/TETC.2014.2371239.[11] C. Wang, N. Cao, J. Li, K. Ren, and W. Lou, “Secure ranked keywordsearch over encrypted cloud data,” in Proceedings of ICDCS. IEEE,2010, pp. 253–262.[12] A. Boldyreva, N. Chenette, Y. Lee, and A. Oneill, “Order-preservingsymmetric encryption,” in Advances in Cryptology-EUROCRYPT.Springer, 2009, pp. 224–241.[13] W. Sun, B. Wang, N. Cao, M. Li, W. Lou, Y. T. Hou, and H. Li,“Verifiable privacy-preserving multi-keyword text search in the cloudsupporting similarity-based ranking,” IEEE Transactions on Parallel andDistributed Systems, vol. DOI: 10.1109/TPDS.2013.282, 2013.[14] J. Yu, P. Lu, Y. Zhu, G. Xue, and M. Li, “Towards secure multikeywordtop-k retrieval over encrypted cloud data,” IEEE Transactionson Dependable and Secure Computing, vol. 10, no. 4, pp. 239–250,2013.[15] A. Arvanitis and G. Koutrika, “Towards preference-aware relationaldatabases,” in International Conference on Data Engineering (ICDE).IEEE, 2012, pp. 426–437.[16] G. Koutrika, E. Pitoura, and K. Stefanidis, “Preference-based querypersonalization,” in Advanced Query Processing. Springer, 2013, pp.57–81.[17] B. Zhang and F. Zhang, “An efficient public key encryption withconjunctive-subset keywords search,” Journal of Network and ComputerApplications, vol. 34, no. 1, pp. 262–267, 2011.[18] D. Stinson, Cryptography: theory and practice. CRC press, 2006.[19] H. Li, D. Liu, K. Jia, and X. Lin, “Achieving authorized and rankedmulti-keyword search over encrypted cloud data,” in Proceedings of ICC.IEEE, 2015, to appear.[20] S. Zerr, E. Demidova, D. Olmedilla, W. Nejdl, M. Winslett, andS. Mitra, “Zerber: r-confidential indexing for distributed documents,” inProceedings of the 11th international conference on Extending databasetechnology: Advances in database technology. ACM, 2008, pp. 287–298.[21] W. K. Wong, D. W.-l. Cheung, B. Kao, and N. Mamoulis, “Secureknn computation on encrypted databases,” in Proceedings of SIGMODInternational Conference on Management of data. ACM, 2009, pp.139–152.[22] J. Zobel and A. Moffat, “Exploring the similarity space,” in ACM SIGIRForum, vol. 32, no. 1. ACM, 1998, pp. 18–34.[23] N. Ferguson, R. Schroeppel, and D. Whiting, “A simple algebraicrepresentation of rijndael,” in Selected Areas in Cryptography. Springer,2001, pp. 103–111.[24] http://kdd.ics.uci.edu/databases/nsfabs/nsfawards.html.1545-5971 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TDSC.2015.2406704, IEEE Transactions on Dependable and Secure Computing14[25] C. Wang, N. Cao, K. Ren, and W. Lou, “Enabling secure and efficientranked keyword search over outsourced cloud data,” IEEE Transactionson Parallel and Distributed Systems, vol. 23, no. 8, pp. 1467–1479,2012.[26] P. Golle, J. Staddon, and B. Waters, “Secure conjunctive keyword searchover encrypted data,” in Applied Cryptography and Network Security.Springer, 2004, pp. 31–45.[27] D. Boneh and B. Waters, “Conjunctive, subset, and range queries onencrypted data,” in Theory of cryptography. Springer, 2007, pp. 535–554.[28] D. Boneh, G. Di Crescenzo, R. Ostrovsky, and G. Persiano, “Public keyencryption with keyword search,” in Advances in Cryptology–Eurocrypt.Springer, 2004, pp. 506–522.[29] Y. Hwang and P. Lee, “Zpublic key encryption with conjunctive keywordsearch and its extension to a multi-user system,” in Proceeding ofPairing. Springer, 2007, pp. 2–22.[30] Q. Liu, C. C. Tan, J. Wu, and G. Wang, “Efficient information retrievalfor ranked queries in cost-effective cloud environments,” in Proceedingsof INFOCOM. IEEE, 2012, pp. 2581–2585.Hongwei Li is an Associate Professor with theSchool of Computer Science and Engineering,University of Electronic Science and Technologyof China, China. He received the PhD degreein computer software and theory from Universityof Electronic Science and Technology of China,China in 2008. He has worked as a Post-Doctoral Fellow in Department of Electrical andComputer Engineering at University of Waterloofor one year until Oct. 2012. His research interestsinclude network security, applied cryptography,and trusted computing. Dr. Li serves as the Associate Editor ofPeer-to-Peer Networking and Applications, the Guest Editor for Peerto-Peer Networking and Applications Special Issue on Security andPrivacy of P2P Networks in Emerging Smart City. He also serves onthe technical program committees for many international conferencessuch as IEEE INFOCOM, IEEE ICC, IEEE GLOBECOM, IEEE WCNC,IEEE SmartGridComm, BODYNETS and IEEE DASC. He is a memberof IEEE, a member of China Computer Federation and a member ofChina Association for Cryptologic Research.Yi Yang received his B.S. degree in NetworkEngineering from Tianjin University of Scienceand Technology (TUST) in 2012. Currently, he isa master student at the School of Computer Scienceand Engineering, University of ElectronicScience and Technology of China (UESTC), China.He serves as the reviewer of Peer-to-PeerNetworking and Application, IEEE INFOCOM,IEEE ICC, IEEE GLOBECOM, IEEE ICCC, etc.His research interests include cryptography, andthe secure smart grid.Tom H. Luan received the B.Sc. degree fromXian Jiaotong University, China, in 2004, theM.Phil. degree from Hong Kong University ofScience and Technology, Hong Kong, China, in2007, and Ph.D. degrees from the Universityof Waterloo, Canada, in 2012. Since December2013, he has been the Lecturer in Mobile andApps at the School of Information Technology,Deakin University, Melbourne Burwood, Australia.His research mainly focuses on vehicularnetworking, wireless content distribution, peerto-peer networking and mobile cloud computing.Xiaohui Liang received the B.Sc. degree inComputer Science and Engineering and theM.Sc. degree in Computer Software and Theoryfrom Shanghai Jiao Tong University (SJTU), China,in 2006 and 2009, respectively. He is currentlyworking toward a Ph.D. degree in the Departmentof Electrical and Computer Engineering,University of Waterloo, Canada. His research interestsinclude applied cryptography, and securityand privacy issues for e-healthcare system,cloud computing, mobile social networks, andsmart grid.Liang Zhou is a professor with the NationalKey Lab of Science and Technology on Communicationat University of Electronic Scienceand Technology of China, China. His currentresearch interests include error control coding,secure communication and cryptography.Xuemin (Sherman) Shen is a Professor andUniversity Research Chair, Department of Electricaland Computer Engineering, University ofWaterloo, Canada. He was the Associate Chairfor Graduate Studies from 2004 to 2008. Dr.Shen’s research focuses on resource managementin interconnected wireless/wired networks,wireless network security, vehicular ad hocand sensor networks. Dr. Shen served as theTechnical Program Committee Chair for IEEEVTC’10 Fall and IEEE Globecom’07. He alsoserves/served as the Editor-in-Chief for IEEE Network, Peer-to-PeerNetworking and Application, and IET Communications; a Founding AreaEditor for IEEE Transactions on Wireless Communications; an AssociateEditor for IEEE Transactions on Vehicular Technology, ComputerNetworks. Dr. Shen is a registered Professional Engineer of Ontario,Canada, an IEEE Fellow, an Engineering Institute of Canada Fellow, aCanadian Academy of Engineering Fellow, and a Distinguished Lecturerof IEEE Vehicular Technology Society and Communications Society.

In mobile cloud computing, a fundamental application is to outsource the mobile data to external cloud servers for scalable data storage. The outsourced data, however, need to be encrypted due to the privacy and confidentiality concerns of their owner. This results in the distinguished difficulties on the accurate search over the encrypted mobile cloud data.

In this paper, we develop the searchable encryption for multi-keyword ranked search over the storage data. Specifically, by considering the large number of outsourced documents (data) in the cloud, we utilize the relevance score and k-nearest neighbor techniques to develop an efficient multi-keyword search scheme that can return the ranked search results based on the accuracy.

This framework, we leverage an efficient index to further improve the search efficiency, and adopt the blind storage system to conceal access pattern of the search user. Security analysis demonstrates that our scheme can achieve confidentiality of documents and index, trapdoor privacy, trapdoor unlinkability, and concealing access pattern of the search user. Finally, using extensive simulations, we show that our proposal can achieve much improved efficiency in terms of search functionality and search time compared with the existing proposals.

1.1 GOAL OF THE PROJECT:

Efficient and privacy-preserving multi-keyword ranked search over encrypted mobile cloud data via blind storage system, the EMRS has following design goals:

• Multi-Keyword Ranked Search: To meet the requirements for practical uses and provide better user experience, the EMRS should not only support multi-keyword search over encrypted mobile cloud data, but also achieve relevance-based result ranking.

• Search Efficiency: Since the number of the total documents may be very large in a practical situation, the EMRS should achieve sublinear search with better search efficiency.

• Confidentiality and Privacy Preservation: To prevent the cloud server from learning any additional information about the documents and the index, and to keep search users’ trapdoors secret, the EMRS should cover all the security requirements that we introduced above.

1.2 INTRODUCTION

Mobile cloud computing gets rid of the hardware limitation of mobile devices by exploring the scalable and virtualized cloud storage and computing resources, and accordingly is able to provide much more powerful and scalable mobile services to users. In mobile cloud computing, mobile users typically outsource their data to external cloud servers, e.g., iCloud, to enjoy a stable, low-cost and scalable way for data storage and access. However, as outsourced data typically contain sensitive privacy information, such as personal photos, emails, etc., which would lead to severe confidentiality and privacy violations, if without efficient protections. It is therefore necessary to encrypt the sensitive data before outsourcing them to the cloud. The data encryption, however, would result in salient difficulties when other users need to access interested data with search, due to the difficulties of search over encrypted data.

This fundamental issue in mobile cloud computing accordingly motivates an extensive body of research in the recent years on the investigation of searchable encryption technique to achieve efficient searching over outsourced encrypted data. A collection of research works have recently been developed on the topic of multi-keyword search over encrypted data. Propose a symmetric searchable encryption scheme which achieves high efficiency for large databases with modest scarification on security guarantees. Propose a multi-keyword search scheme supporting result ranking by adopting k-nearest neighbors (kNN) technique. Propose a dynamic searchable encryption scheme through blind storage to conceal access pattern of the search user.

In order to meet the practical search requirements, search over encrypted data should support the following three functions.

First, the searchable encryption schemes should support multi-keyword search, and provide the same user experience as searching in Google search with different keywords; single-keyword search is far from satisfactory by only returning very limited and inaccurate search results. Second, to quickly identify most relevant results, the search user would typically prefer cloud servers to sort the returned search results in a relevance-based order ranked by the relevance of the search request to the documents. In addition, showing the ranked search to users can also eliminate the unnecessary network traffic by only sending back the most relevant results from cloud to search users.

Third, as for the search efficiency, since the number of the documents contained in a database could be extraordinarily large, searchable encryption schemes should be efficient to quickly respond to the search requests with minimum delays.

In contrast to the theoretical benefits, most of the existing proposals, however, fail to offer sufficient insights towards the construction of full functioned searchable encryption as described above. As an effort towards the issue, in this paper, we propose an efficient multi-keyword ranked search (EMRS) scheme over encrypted mobile cloud data through blind storage.

Our main contributions can be summarized as follows:

• We introduce a relevance score in searchable encryption to achieve multi-keyword ranked search over the encrypted mobile cloud data. In addition to that, we construct an efficient index to improve the search efficiency.

• By modifying the blind storage system in the EMRS, we solve the trapdoor unlinkability problem and conceal access pattern of the search user from the cloud server.

• We give thorough security analysis to demonstrate that the EMRS can reach a high security level including confidentiality of documents and index, trapdoor privacy, trapdoor unlinkability, and concealing access pattern of the search user. Moreover, we implement extensive experiments, which show that the EMRS can achieve enhanced efficiency in the terms of functionality and search efficiency compared with existing proposals.

1.3 LITRATURE SURVEY

CHAPTER 2

2.0 SYSTEM ANALYSIS

2.1 EXISTING SYSTEM:

Existing works built various types of secure index and corresponding index-based keyword matching algorithms to improve search efficiency. All these works only support the search of single keyword. Subsequent works extended the search capability to multiple, conjunctive or disjunctive, keywords search. However, they support only exact keyword matching. Misspelled keywords in the query will result in wrong or no matching. Very recently, a few works extended the search capability to approximate keyword matching (also known as fuzzy search). These are all for single keyword search, with a common approach involving expanding the index file by covering possible combinations of keyword misspelling so that a certain degree of spelling error, measured by edit distance, can be tolerated. Although a wild-card approach is adopted to minimize the expansion of the resulting index file, for a l-letter long keyword to tolerate an error up to an edit distance of d, the index has to be expanded times.

Thus, it is not scalable as the storage complexity increases exponentially with the increase of the error tolerance. To support multi-keyword search, the search algorithm will have to run multiple rounds To date, efficient multi-keyword fuzzy search over encrypted data remains a challenging problem. We want to point out that the efforts on search over encrypted data involve not only information retrieval techniques such as advanced data structures used to represent the searchable index, and efficient search algorithms that run over the corresponding data structure, but also the proper design of cryptographic protocols to ensure the security and privacy of the overall system. Although single keyword search and fuzzy search have been implemented separately, a combination of the two does not lead to a secure and efficient single keyword fuzzy search scheme.

2.1.1 DISADVANTAGES:

The large number of data users and documents in cloud, it is crucial for the search service to allow multi-keyword query and provide result similarity ranking to meet the effective data retrieval need. The searchable encryption focuses on single keyword search or Boolean keyword search, and rarely differentiates the search results.

• Single-keyword search without ranking
• Boolean- keyword search without ranking
• Single-keyword similarity search with ranking

2.2 PROPOSED SYSTEM:

Propose a symmetric searchable encryption scheme which achieves high efficiency for large databases with modest scarification on security guarantees. Propose a multi-keyword search scheme supporting result ranking by adopting k-nearest neighbors (kNN) technique. Propose a dynamic searchable encryption scheme through blind storage to conceal access pattern of the search user.

We propose the detailed EMRS. Since the encrypted documents and index z are both stored in the blind storage system, we would provide the general construction of the blind storage system. Moreover, since the EMRS aims to eliminate the risk of sharing the key that is used to encrypt the documents with all search users and solve the trapdoor unlinkability problem in Naveed’s scheme.

We modify the construction of blind storage and leverage ciphertext policy attribute-based encryption (CP-ABE) technique in the EMRS. However, specific construction of CP-ABE is out of scope of this paper and we only give a simple indication here. The notations of this paper are shown in Table 1. The EMRS consists of the following phases: System Setup, Construction of Blind Storage, Encrypted Database Setup, Trapdoor Generation, Efficient and Secure Search, and Retrieve Documents from Blind Storage.

2.2.1 ADVANTAGES:

In this paper, we propose an efficient multi-keyword ranked search (EMRS) scheme over encrypted mobile cloud data through blind storage.

Our main contributions can be summarized as follows:

• We introduce a relevance score in searchable encryption to achieve multi-keyword ranked search over the encrypted mobile cloud data. In addition to that, we construct an efficient index to improve the search efficiency.

• By modifying the blind storage system in the EMRS, we solve the trapdoor unlinkability problem and conceal access pattern of the search user from the cloud server.

• We give thorough security analysis to demonstrate that the EMRS can reach a high security level including confidentiality of documents and index, trapdoor privacy, trapdoor unlinkability, and concealing access pattern of the search user. Moreover, we implement extensive experiments, which show that the EMRS can achieve enhanced efficiency in the terms of functionality and search efficiency compared with existing proposals

2.3 HARDWARE & SOFTWARE REQUIREMENTS:

2.3.1 HARDWARE REQUIREMENT:

v    Processor                                 –    Pentium –IV

• Speed                                      –    1.1 GHz
  • RAM                                       –    256 MB (min)
  • Hard Disk                               –   20 GB
  • Floppy Drive                           –    1.44 MB
  • Key Board                              –    Standard Windows Keyboard
  • Mouse                                     –    Two or Three Button Mouse
  • Monitor                                   –    SVGA

 

2.3.2 SOFTWARE REQUIREMENTS:

• Operating System                   :           Windows XP or Win7
• Front End                                :           JAVA JDK 1.7
• Back End                                :           MYSQL Server
• Server                                      :           Apache Tomact Server
• Script                                       :           JSP Script
• Document                               :           MS-Office 2007

CHAPTER 3

3.0 SYSTEM DESIGN:

Data Flow Diagram / Use Case Diagram / Flow Diagram:

• The DFD is also called as bubble chart. It is a simple graphical formalism that can be used to represent a system in terms of the input data to the system, various processing carried out on these data, and the output data is generated by the system

• The data flow diagram (DFD) is one of the most important modeling tools. It is used to model the system components. These components are the system process, the data used by the process, an external entity that interacts with the system and the information flows in the system.

• DFD shows how the information moves through the system and how it is modified by a series of transformations. It is a graphical technique that depicts information flow and the transformations that are applied as data moves from input to output.

• DFD is also known as bubble chart. A DFD may be used to represent a system at any level of abstraction. DFD may be partitioned into levels that represent increasing information flow and functional detail.

NOTATION:

SOURCE OR DESTINATION OF DATA:

External sources or destinations, which may be people or organizations or other entities

DATA SOURCE:

Here the data referenced by a process is stored and retrieved.

PROCESS:

People, procedures or devices that produce data’s in the physical component is not identified.

DATA FLOW:

Data moves in a specific direction from an origin to a destination. The data flow is a “packet” of data.

MODELING RULES:

There are several common modeling rules when creating DFDs:

1. All processes must have at least one data flow in and one data flow out.
2. All processes should modify the incoming data, producing new forms of outgoing data.
3. Each data store must be involved with at least one data flow.
4. Each external entity must be involved with at least one data flow.
5. A data flow must be attached to at least one process.

3.1 ARCHITECTURE DIAGRAM

3.2 DATAFLOW DIAGRAM

UML DIAGRAMS:

3.2 USE CASE DIAGRAM:

3.3 CLASS DIAGRAM:

3.4 SEQUENCE DIAGRAM:

3.5 ACTIVITY DIAGRAM:

CHAPTER 4

4.0 IMPLEMENTATION:

EMRS SCHEME (EFFICIENT MULTI-KEYWORD RANKED SEARCH):

4.1 ALGORITHM

CP-ABE ENCRYPTION ALGORITHM:

The data owner builds the encrypted database as follows:

Step 1: The data owner computes the d-dimension relevance vector p = (p1, p2, · · · pd ) for each document using the TF-IDF weighting technique, where pj for j ∈ (1, 2 · · · d) represents the weighting of keyword ωj in document di . Then, the data owner extends the p to a (d+2)-dimension vector p ∗ . The (d+1)-th entry of p ∗ is set to a random number ε and the (d+2)-th entry is set to 1. We would let ε follow a normal distribution N(µ, σ2 ) [11]. For each document di , to compute the encrypted relevance vector, the data owner encrypts the associated extended relevance vector p ∗ using the secret key M1, M2 and S. First, the data owner chooses a random number r and splits the extended relevance vector p ∗ into two (d+2)-dimension vectors p 0 and p 00 using the vector S. For the j-th item in p ∗ , set

Step 2: For each document di in D, set the document into blocks of mb bits each. For each block, there is a header H(idi) indicating that this block belongs to document di . And the sizei of the document is contained in the header of the first block of di . Then, for each document di , the data owner chooses a 192-bit key Ki for the algorithm Enc(). More precisely, for each block B[j] of the document di , where j represents the index number of this block, compute the Ki ⊕ 8(j) as the key for the encryption of this block. Since each block has a unique index number, the blocks of the same document are encrypted with different keys. The document di contains sizei encrypted blocks and the first block of the document di with index number j is as

Finally, the data owner encrypts all the documents and writes them to the blind storage system using the B.Build function. Step 3: To enable efficient search over the encrypted documents, the data owner builds the index z. First, the data owner defines the access policy υi for each document di . We denote the result of attribute-based encryption using access policy υi as ABEυi (). The data owner initializes z to an empty array indexed by all keywords. Then, the index z can be constructed as shown in Algorithm 1.

4.2 MODULES:

SEARCHABLE ENCRYPTION CP-ABE:

MULTI-KEYWORD RANKED SEARCH:

BLIND STORAGE SYSTEM:

EMRS SECURITY REQUIRMENTS:

4.3 MODULE DESCRIPTION:

SEARCHABLE ENCRYPTION CP-ABE:

In ciphertext policy attribute-based encryption (CP-ABE), ciphertexts are created with an access structure (usually an access tree) which defines the access policy.A user can decrypt the data only if the attributes embedded in his attribute keys satisfy the access policy in the ciphertext. In CP-ABE, the encrypter holds the ultimate authority of the access policy. The documents are encrypted by the traditional symmetric cryptography technique before being outsourced to the cloud server. Without a correct key, the search user and cloud server cannot decrypt the documents. As for index confidentiality, the relevance vector for each document is encrypted using the secret key M1, M2, and S. And the descriptors of the documents are encrypted using CP-ABE technique. Thus, the cloud server can only use the index z to retrieve the encrypted relevance vectors without knowing any additional information, such as the associations between the documents and the keywords. And only the search user with correct attribute keys can decrypt the descriptor ABE_i (idijjKijjx) to get the document id and the associated symmetric key. Thus, the confidentiality of documents and index can be well protected.

MULTI-KEYWORD RANKED SEARCH:

Multi-keyword rankedsearch over encrypted data should support the following three functions. First, the searchable encryption schemes should support multi-keyword search, and provide the same user experience as searching in Google search with different keywords; single-keyword search is far from satisfactory by only returning very limited and inaccurate search results. Second, to quickly identify most relevant results, the search user would typically prefer cloud servers to sort the returned search results in a relevance-based order ranked by the relevance of the search request to the documents. In addition, showing the ranked search to users can also eliminate the unnecessary network traffic by only sending back the most relevant results from cloud to search users. Third, as for the search efficiency, since the number of the documents contained in a database could be extraordinarily large, searchable encryption schemes should be efficient to quickly respond to the search requests with minimum delays.

BLIND STORAGE SYSTEM:

A blind storage system is built on the cloud server to support adding, updating and deleting documents and concealing the access pattern of the search user from the cloud server. In the blind storage system, all documents are divided into fixed-size blocks. These blocks are indexed by a sequence of random integers generated by a document-related seed. In the view of a cloud server, it can only see the blocks of encrypted documents uploaded and downloaded. Thus, the blind storage system leaks little information to the cloud server. Specifically, the cloud server does not know which blocks are of the same document, even the total number of the documents and the size of each document. Moreover, all the documents and index can be stored in the blind storage system to achieve a searchable encryption scheme.

EMRS SECURITY REQUIRMENTS:

EMRS, we consider the cloud server to be curious but honest which means it executes the task assigned by the data owner and the search user correctly. However, it is curious about the data in its storage and the received trapdoors to obtain additional information. Moreover, we consider the Knowing Background model in the EMRS, which allows the cloud server to know more background information of the documents such as statistical information of the keywords.

Specifically, the EMRS aims to provide the following four security requirements:

• Confidentiality of Documents and Index: Documents and index should be encrypted before being outsourced to a cloud server. The cloud server should be prevented from prying into the outsourced documents and cannot deduce any associations between the documents and keywords using the index.

• Trapdoor Privacy: Since the search user would like to keep her searches from being exposed to the cloud server, the cloud server should be prevented from knowing the exact keywords contained in the trapdoor of the search user.

• Trapdoor Unlinkability: The trapdoors should not be linkable, which means the trapdoors should be totally different even if they contain the same keywords. In other words, the trapdoors should be randomized rather than determined. The cloud server cannot deduce any associations between two trapdoors.

• Concealing Access Pattern of the Search User: Access pattern is the sequence of the searched results. In the EMRS, the access pattern should be totally concealed from the cloud server. Specifically, the cloud server cannot learn the total number of the documents stored on it nor the size of the searched document even when the search user retrieves this document from the cloud server.

CHAPTER 5

5.0 SYSTEM STUDY:

5.1 FEASIBILITY STUDY:

The feasibility of the project is analyzed in this phase and business proposal is put forth with a very general plan for the project and some cost estimates. During system analysis the feasibility study of the proposed system is to be carried out. This is to ensure that the proposed system is not a burden to the company.  For feasibility analysis, some understanding of the major requirements for the system is essential.

Three key considerations involved in the feasibility analysis are 

• ECONOMICAL FEASIBILITY
• TECHNICAL FEASIBILITY
• SOCIAL FEASIBILITY

5.1.1 ECONOMICAL FEASIBILITY:     

This study is carried out to check the economic impact that the system will have on the organization. The amount of fund that the company can pour into the research and development of the system is limited. The expenditures must be justified. Thus the developed system as well within the budget and this was achieved because most of the technologies used are freely available. Only the customized products had to be purchased.

 

5.1.2 TECHNICAL FEASIBILITY   

This study is carried out to check the technical feasibility, that is, the technical requirements of the system. Any system developed must not have a high demand on the available technical resources. This will lead to high demands on the available technical resources. This will lead to high demands being placed on the client. The developed system must have a modest requirement, as only minimal or null changes are required for implementing this system.  

5.1.3 SOCIAL FEASIBILITY:  

The aspect of study is to check the level of acceptance of the system by the user. This includes the process of training the user to use the system efficiently. The user must not feel threatened by the system, instead must accept it as a necessity. The level of acceptance by the users solely depends on the methods that are employed to educate the user about the system and to make him familiar with it. His level of confidence must be raised so that he is also able to make some constructive criticism, which is welcomed, as he is the final user of the system.

5.2 SYSTEM TESTING:

Testing is a process of checking whether the developed system is working according to the original objectives and requirements. It is a set of activities that can be planned in advance and conducted systematically. Testing is vital to the success of the system. System testing makes a logical assumption that if all the parts of the system are correct, the global will be successfully achieved. In adequate testing if not testing leads to errors that may not appear even many months.

This creates two problems, the time lag between the cause and the appearance of the problem and the effect of the system errors on the files and records within the system. A small system error can conceivably explode into a much larger Problem. Effective testing early in the purpose translates directly into long term cost savings from a reduced number of errors. Another reason for system testing is its utility, as a user-oriented vehicle before implementation. The best programs are worthless if it produces the correct outputs.

5.2.1 UNIT TESTING:

Description	Expected result
Test for application window properties.	All the properties of the windows are to be properly aligned and displayed.
Test for mouse operations.	All the mouse operations like click, drag, etc. must perform the necessary operations without any exceptions.

A program represents the logical elements of a system. For a program to run satisfactorily, it must compile and test data correctly and tie in properly with other programs. Achieving an error free program is the responsibility of the programmer. Program  testing  checks  for  two  types  of  errors:  syntax  and  logical. Syntax error is a program statement that violates one or more rules of the language in which it is written. An improperly defined field dimension or omitted keywords are common syntax errors. These errors are shown through error message generated by the computer. For Logic errors the programmer must examine the output carefully.

5.1.2 FUNCTIONAL TESTING:

Functional testing of an application is used to prove the application delivers correct results, using enough inputs to give an adequate level of confidence that will work correctly for all sets of inputs. The functional testing will need to prove that the application works for each client type and that personalization function work correctly.When a program is tested, the actual output is compared with the expected output. When there is a discrepancy the sequence of instructions must be traced to determine the problem.  The process is facilitated by breaking the program into self-contained portions, each of which can be checked at certain key points. The idea is to compare program values against desk-calculated values to isolate the problems.

Description	Expected result
Test for all modules.	All peers should communicate in the group.
Test for various peer in a distributed network framework as it display all users available in the group.	The result after execution should give the accurate result.

5.1. 3 NON-FUNCTIONAL TESTING:

 The Non Functional software testing encompasses a rich spectrum of testing strategies, describing the expected results for every test case. It uses symbolic analysis techniques. This testing used to check that an application will work in the operational environment. Non-functional testing includes:

• Load testing
• Performance testing
• Usability testing
• Reliability testing
• Security testing

5.1.4 LOAD TESTING:

An important tool for implementing system tests is a Load generator. A Load generator is essential for testing quality requirements such as performance and stress. A load can be a real load, that is, the system can be put under test to real usage by having actual telephone users connected to it. They will generate test input data for system test.

Description	Expected result
It is necessary to ascertain that the application behaves correctly under loads when ‘Server busy’ response is received.	Should designate another active node as a Server.

5.1.5 PERFORMANCE TESTING:

Performance tests are utilized in order to determine the widely defined performance of the software system such as execution time associated with various parts of the code, response time and device utilization. The intent of this testing is to identify weak points of the software system and quantify its shortcomings.

Description	Expected result
This is required to assure that an application perforce adequately, having the capability to handle many peers, delivering its results in expected time and using an acceptable level of resource and it is an aspect of operational management.	Should handle large input values, and produce accurate result in a  expected time.

5.1.6 RELIABILITY TESTING:

The software reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time and it is being ensured in this testing. Reliability can be expressed as the ability of the software to reveal defects under testing conditions, according to the specified requirements. It the portability that a software system will operate without failure under given conditions for a given time interval and it focuses on the behavior of the software element. It forms a part of the software quality control team.

Description	Expected result
This is to check that the server is rugged and reliable and can handle the failure of any of the components involved in provide the application.	In case of failure of  the server an alternate server should take over the job.

5.1.7 SECURITY TESTING:

Security testing evaluates system characteristics that relate to the availability, integrity and confidentiality of the system data and services. Users/Clients should be encouraged to make sure their security needs are very clearly known at requirements time, so that the security issues can be addressed by the designers and testers.

Description	Expected result
Checking that the user identification is authenticated.	In case failure it should not be connected in the framework.
Check whether group keys in a tree are shared by all peers.	The peers should know group key in the same group.

5.1.8 WHITE BOX TESTING:

White  box  testing,  sometimes called  glass-box  testing is  a test  case  design method  that  uses  the  control  structure  of the procedural  design  to  derive  test  cases. Using  white  box  testing  method,  the software  engineer  can  derive  test  cases. The White box testing focuses on the inner structure of the software structure to be tested.

Description	Expected result
Exercise all logical decisions on their true and false sides.	All the logical decisions must be valid.
Execute all loops at their boundaries and within their operational bounds.	All the loops must be finite.
Exercise internal data structures to ensure their validity.	All the data structures must be valid.

5.1.9 BLACK BOX TESTING:

Black box testing, also called behavioral testing, focuses on the functional requirements of the software.  That  is,  black  testing  enables  the software engineer  to  derive  sets  of  input  conditions  that  will  fully  exercise  all  functional requirements  for  a  program.  Black box testing is not alternative to white box techniques.  Rather  it  is  a  complementary  approach  that  is  likely  to  uncover  a different  class  of  errors  than  white box  methods. Black box testing attempts to find errors which focuses on inputs, outputs, and principle function of a software module. The starting point of the black box testing is either a specification or code. The contents of the box are hidden and the stimulated software should produce the desired results.

Description	Expected result
To check for incorrect or missing functions.	All the functions must be valid.
To check for interface errors.	The entire interface must function normally.
To check for errors in a data structures or external data base access.	The database updation and retrieval must be done.
To check for initialization and termination errors.	All the functions and data structures must be initialized properly and terminated normally.

All the above system testing strategies are carried out in as the development, documentation and institutionalization of the proposed goals and related policies is essential.

CHAPTER 6

6.0 SOFTWARE DESCRIPTION:

 

6.1 JAVA TECHNOLOGY:

Java technology is both a programming language and a platform.

 

The Java Programming Language

 

The Java programming language is a high-level language that can be characterized by all of the following buzzwords:

• Simple
  • Architecture neutral
  • Object oriented
  • Portable
  • Distributed     
  • High performance
  • Interpreted     
  • Multithreaded
  • Robust
  • Dynamic
  • Secure     

With most programming languages, you either compile or interpret a program so that you can run it on your computer. The Java programming language is unusual in that a program is both compiled and interpreted. With the compiler, first you translate a program into an intermediate language called Java byte codes —the platform-independent codes interpreted by the interpreter on the Java platform. The interpreter parses and runs each Java byte code instruction on the computer. Compilation happens just once; interpretation occurs each time the program is executed. The following figure illustrates how this works.

You can think of Java byte codes as the machine code instructions for the Java Virtual Machine (Java VM). Every Java interpreter, whether it’s a development tool or a Web browser that can run applets, is an implementation of the Java VM. Java byte codes help make “write once, run anywhere” possible. You can compile your program into byte codes on any platform that has a Java compiler. The byte codes can then be run on any implementation of the Java VM. That means that as long as a computer has a Java VM, the same program written in the Java programming language can run on Windows 2000, a Solaris workstation, or on an iMac.

6.2 THE JAVA PLATFORM:

A platform is the hardware or software environment in which a program runs. We’ve already mentioned some of the most popular platforms like Windows 2000, Linux, Solaris, and MacOS. Most platforms can be described as a combination of the operating system and hardware. The Java platform differs from most other platforms in that it’s a software-only platform that runs on top of other hardware-based platforms.

The Java platform has two components:

• The Java Virtual Machine (Java VM)
• The Java Application Programming Interface (Java API)

You’ve already been introduced to the Java VM. It’s the base for the Java platform and is ported onto various hardware-based platforms.

The Java API is a large collection of ready-made software components that provide many useful capabilities, such as graphical user interface (GUI) widgets. The Java API is grouped into libraries of related classes and interfaces; these libraries are known as packages. The next section, What Can Java Technology Do? Highlights what functionality some of the packages in the Java API provide.

The following figure depicts a program that’s running on the Java platform. As the figure shows, the Java API and the virtual machine insulate the program from the hardware.

Native code is code that after you compile it, the compiled code runs on a specific hardware platform. As a platform-independent environment, the Java platform can be a bit slower than native code. However, smart compilers, well-tuned interpreters, and just-in-time byte code compilers can bring performance close to that of native code without threatening portability.

6.3 WHAT CAN JAVA TECHNOLOGY DO?

The most common types of programs written in the Java programming language are applets and applications. If you’ve surfed the Web, you’re probably already familiar with applets. An applet is a program that adheres to certain conventions that allow it to run within a Java-enabled browser.

However, the Java programming language is not just for writing cute, entertaining applets for the Web. The general-purpose, high-level Java programming language is also a powerful software platform. Using the generous API, you can write many types of programs.

An application is a standalone program that runs directly on the Java platform. A special kind of application known as a server serves and supports clients on a network. Examples of servers are Web servers, proxy servers, mail servers, and print servers. Another specialized program is a servlet.

A servlet can almost be thought of as an applet that runs on the server side. Java Servlets are a popular choice for building interactive web applications, replacing the use of CGI scripts. Servlets are similar to applets in that they are runtime extensions of applications. Instead of working in browsers, though, servlets run within Java Web servers, configuring or tailoring the server.

How does the API support all these kinds of programs? It does so with packages of software components that provides a wide range of functionality. Every full implementation of the Java platform gives you the following features:

• The essentials: Objects, strings, threads, numbers, input and output, data structures, system properties, date and time, and so on.
• Applets: The set of conventions used by applets.
• Networking: URLs, TCP (Transmission Control Protocol), UDP (User Data gram Protocol) sockets, and IP (Internet Protocol) addresses.
• Internationalization: Help for writing programs that can be localized for users worldwide. Programs can automatically adapt to specific locales and be displayed in the appropriate language.
• Security: Both low level and high level, including electronic signatures, public and private key management, access control, and certificates.
• Software components: Known as JavaBeans^TM, can plug into existing component architectures.
• Object serialization: Allows lightweight persistence and communication via Remote Method Invocation (RMI).
• Java Database Connectivity (JDBC^TM): Provides uniform access to a wide range of relational databases.

The Java platform also has APIs for 2D and 3D graphics, accessibility, servers, collaboration, telephony, speech, animation, and more. The following figure depicts what is included in the Java 2 SDK.

 

6.4 HOW WILL JAVA TECHNOLOGY CHANGE MY LIFE?

We can’t promise you fame, fortune, or even a job if you learn the Java programming language. Still, it is likely to make your programs better and requires less effort than other languages. We believe that Java technology will help you do the following:

• Get started quickly: Although the Java programming language is a powerful object-oriented language, it’s easy to learn, especially for programmers already familiar with C or C++.
• Write less code: Comparisons of program metrics (class counts, method counts, and so on) suggest that a program written in the Java programming language can be four times smaller than the same program in C++.
• Write better code: The Java programming language encourages good coding practices, and its garbage collection helps you avoid memory leaks. Its object orientation, its JavaBeans component architecture, and its wide-ranging, easily extendible API let you reuse other people’s tested code and introduce fewer bugs.
• Develop programs more quickly: Your development time may be as much as twice as fast versus writing the same program in C++. Why? You write fewer lines of code and it is a simpler programming language than C++.
• Avoid platform dependencies with 100% Pure Java: You can keep your program portable by avoiding the use of libraries written in other languages. The 100% Pure Java^TM Product Certification Program has a repository of historical process manuals, white papers, brochures, and similar materials online.
• Write once, run anywhere: Because 100% Pure Java programs are compiled into machine-independent byte codes, they run consistently on any Java platform.
• Distribute software more easily: You can upgrade applets easily from a central server. Applets take advantage of the feature of allowing new classes to be loaded “on the fly,” without recompiling the entire program.

 

6.5 ODBC:

 

Microsoft Open Database Connectivity (ODBC) is a standard programming interface for application developers and database systems providers. Before ODBC became a de facto standard for Windows programs to interface with database systems, programmers had to use proprietary languages for each database they wanted to connect to. Now, ODBC has made the choice of the database system almost irrelevant from a coding perspective, which is as it should be. Application developers have much more important things to worry about than the syntax that is needed to port their program from one database to another when business needs suddenly change.

Through the ODBC Administrator in Control Panel, you can specify the particular database that is associated with a data source that an ODBC application program is written to use. Think of an ODBC data source as a door with a name on it. Each door will lead you to a particular database. For example, the data source named Sales Figures might be a SQL Server database, whereas the Accounts Payable data source could refer to an Access database. The physical database referred to by a data source can reside anywhere on the LAN.

The ODBC system files are not installed on your system by Windows 95. Rather, they are installed when you setup a separate database application, such as SQL Server Client or Visual Basic 4.0. When the ODBC icon is installed in Control Panel, it uses a file called ODBCINST.DLL. It is also possible to administer your ODBC data sources through a stand-alone program called ODBCADM.EXE. There is a 16-bit and a 32-bit version of this program and each maintains a separate list of ODBC data sources.

From a programming perspective, the beauty of ODBC is that the application can be written to use the same set of function calls to interface with any data source, regardless of the database vendor. The source code of the application doesn’t change whether it talks to Oracle or SQL Server. We only mention these two as an example. There are ODBC drivers available for several dozen popular database systems. Even Excel spreadsheets and plain text files can be turned into data sources. The operating system uses the Registry information written by ODBC Administrator to determine which low-level ODBC drivers are needed to talk to the data source (such as the interface to Oracle or SQL Server). The loading of the ODBC drivers is transparent to the ODBC application program. In a client/server environment, the ODBC API even handles many of the network issues for the application programmer.

The advantages of this scheme are so numerous that you are probably thinking there must be some catch. The only disadvantage of ODBC is that it isn’t as efficient as talking directly to the native database interface. ODBC has had many detractors make the charge that it is too slow. Microsoft has always claimed that the critical factor in performance is the quality of the driver software that is used. In our humble opinion, this is true. The availability of good ODBC drivers has improved a great deal recently. And anyway, the criticism about performance is somewhat analogous to those who said that compilers would never match the speed of pure assembly language. Maybe not, but the compiler (or ODBC) gives you the opportunity to write cleaner programs, which means you finish sooner. Meanwhile, computers get faster every year.

6.6 JDBC:

In an effort to set an independent database standard API for Java; Sun Microsystems developed Java Database Connectivity, or JDBC. JDBC offers a generic SQL database access mechanism that provides a consistent interface to a variety of RDBMSs. This consistent interface is achieved through the use of “plug-in” database connectivity modules, or drivers. If a database vendor wishes to have JDBC support, he or she must provide the driver for each platform that the database and Java run on.

To gain a wider acceptance of JDBC, Sun based JDBC’s framework on ODBC. As you discovered earlier in this chapter, ODBC has widespread support on a variety of platforms. Basing JDBC on ODBC will allow vendors to bring JDBC drivers to market much faster than developing a completely new connectivity solution.

JDBC was announced in March of 1996. It was released for a 90 day public review that ended June 8, 1996. Because of user input, the final JDBC v1.0 specification was released soon after.

The remainder of this section will cover enough information about JDBC for you to know what it is about and how to use it effectively. This is by no means a complete overview of JDBC. That would fill an entire book.

 

6.7 JDBC Goals:

Few software packages are designed without goals in mind. JDBC is one that, because of its many goals, drove the development of the API. These goals, in conjunction with early reviewer feedback, have finalized the JDBC class library into a solid framework for building database applications in Java.

The goals that were set for JDBC are important. They will give you some insight as to why certain classes and functionalities behave the way they do. The eight design goals for JDBC are as follows:

SQL Level API

The designers felt that their main goal was to define a SQL interface for Java. Although not the lowest database interface level possible, it is at a low enough level for higher-level tools and APIs to be created. Conversely, it is at a high enough level for application programmers to use it confidently. Attaining this goal allows for future tool vendors to “generate” JDBC code and to hide many of JDBC’s complexities from the end user.

SQL Conformance

SQL syntax varies as you move from database vendor to database vendor. In an effort to support a wide variety of vendors, JDBC will allow any query statement to be passed through it to the underlying database driver. This allows the connectivity module to handle non-standard functionality in a manner that is suitable for its users.

JDBC must be implemental on top of common database interfaces

The JDBC SQL API must “sit” on top of other common SQL level APIs. This goal allows JDBC to use existing ODBC level drivers by the use of a software interface. This interface would translate JDBC calls to ODBC and vice versa.

1. Provide a Java interface that is consistent with the rest of the Java system

Because of Java’s acceptance in the user community thus far, the designers feel that they should not stray from the current design of the core Java system.

• Keep it simple

This goal probably appears in all software design goal listings. JDBC is no exception. Sun felt that the design of JDBC should be very simple, allowing for only one method of completing a task per mechanism. Allowing duplicate functionality only serves to confuse the users of the API.

• Use strong, static typing wherever possible

Strong typing allows for more error checking to be done at compile time; also, less error appear at runtime.

• Keep the common cases simple

Because more often than not, the usual SQL calls used by the programmer are simple SELECT’s, INSERT’s, DELETE’s and UPDATE’s, these queries should be simple to perform with JDBC. However, more complex SQL statements should also be possible.

Finally we decided to precede the implementation using Java Networking.

And for dynamically updating the cache table we go for MS Access database.

Java ha two things: a programming language and a platform.

Java is a high-level programming language that is all of the following

Simple                                     Architecture-neutral

Object-oriented                       Portable

Distributed                              High-performance

Interpreted                              Multithreaded

Robust                                     Dynamic Secure

Java is also unusual in that each Java program is both compiled and interpreted. With a compile you translate a Java program into an intermediate language called Java byte codes the platform-independent code instruction is passed and run on the computer.

Compilation happens just once; interpretation occurs each time the program is executed. The figure illustrates how this works.

Java Program

Compilers

Interpreter

My Program

6.7 NETWORKING TCP/IP STACK:

The TCP/IP stack is shorter than the OSI one:

TCP is a connection-oriented protocol; UDP (User Datagram Protocol) is a connectionless protocol.

IP datagram’s:

The IP layer provides a connectionless and unreliable delivery system. It considers each datagram independently of the others. Any association between datagram must be supplied by the higher layers. The IP layer supplies a checksum that includes its own header. The header includes the source and destination addresses. The IP layer handles routing through an Internet. It is also responsible for breaking up large datagram into smaller ones for transmission and reassembling them at the other end.

UDP:

UDP is also connectionless and unreliable. What it adds to IP is a checksum for the contents of the datagram and port numbers. These are used to give a client/server model – see later.

TCP:

TCP supplies logic to give a reliable connection-oriented protocol above IP. It provides a virtual circuit that two processes can use to communicate.

Internet addresses

In order to use a service, you must be able to find it. The Internet uses an address scheme for machines so that they can be located. The address is a 32 bit integer which gives the IP address.

Network address:

Class A uses 8 bits for the network address with 24 bits left over for other addressing. Class B uses 16 bit network addressing. Class C uses 24 bit network addressing and class D uses all 32.

Subnet address:

Internally, the UNIX network is divided into sub networks. Building 11 is currently on one sub network and uses 10-bit addressing, allowing 1024 different hosts.

Host address:

8 bits are finally used for host addresses within our subnet. This places a limit of 256 machines that can be on the subnet.

Total address:

The 32 bit address is usually written as 4 integers separated by dots.

Port addresses

A service exists on a host, and is identified by its port. This is a 16 bit number. To send a message to a server, you send it to the port for that service of the host that it is running on. This is not location transparency! Certain of these ports are “well known”.

Sockets:

A socket is a data structure maintained by the system to handle network connections. A socket is created using the call socket. It returns an integer that is like a file descriptor. In fact, under Windows, this handle can be used with Read File and Write File functions.

#include <sys/types.h>

#include <sys/socket.h>

int socket(int family, int type, int protocol);

Here “family” will be AF_INET for IP communications, protocol will be zero, and type will depend on whether TCP or UDP is used. Two processes wishing to communicate over a network create a socket each. These are similar to two ends of a pipe – but the actual pipe does not yet exist.

6.8 JFREE CHART:

JFreeChart is a free 100% Java chart library that makes it easy for developers to display professional quality charts in their applications. JFreeChart’s extensive feature set includes:

A consistent and well-documented API, supporting a wide range of chart types;

A flexible design that is easy to extend, and targets both server-side and client-side applications;

Support for many output types, including Swing components, image files (including PNG and JPEG), and vector graphics file formats (including PDF, EPS and SVG);

JFreeChart is “open source” or, more specifically, free software. It is distributed under the terms of the GNU Lesser General Public Licence (LGPL), which permits use in proprietary applications.

 

6.8.1. Map Visualizations:

Charts showing values that relate to geographical areas. Some examples include: (a) population density in each state of the United States, (b) income per capita for each country in Europe, (c) life expectancy in each country of the world. The tasks in this project include: Sourcing freely redistributable vector outlines for the countries of the world, states/provinces in particular countries (USA in particular, but also other areas);

Creating an appropriate dataset interface (plus default implementation), a rendered, and integrating this with the existing XYPlot class in JFreeChart; Testing, documenting, testing some more, documenting some more.

6.8.2. Time Series Chart Interactivity

Implement a new (to JFreeChart) feature for interactive time series charts — to display a separate control that shows a small version of ALL the time series data, with a sliding “view” rectangle that allows you to select the subset of the time series data to display in the main chart.

6.8.3. Dashboards

There is currently a lot of interest in dashboard displays. Create a flexible dashboard mechanism that supports a subset of JFreeChart chart types (dials, pies, thermometers, bars, and lines/time series) that can be delivered easily via both Java Web Start and an applet.

 

6.8.4. Property Editors

The property editor mechanism in JFreeChart only handles a small subset of the properties that can be set for charts. Extend (or reimplement) this mechanism to provide greater end-user control over the appearance of the charts.

CHAPTER 7

7.0 APPENDIX

7.1 SAMPLE SCREEN SHOTS:

7.2 SAMPLE SOURCE CODE:

CHAPTER 8

8.1 CONCLUSION

In this paper, we have proposed a multi-keyword ranked search scheme to enable accurate, efficient and secure search over encrypted mobile cloud data. Security analysis have demonstrated that proposed scheme can effectively achieve confidentiality of documents and index, trapdoor privacy, trapdoor unlinkability, and concealing access pattern of the search user. Extensive performance evaluations have shown that the proposed scheme can achieve better efficiency in terms of the functionality and computation overhead compared with existing ones. For the future work, we will investigate on the authentication and access control issues in searchable encryption technique.

Abstract—Graph-based ranking models have been widely applied in information retrieval area. In this paper, we focus on a wellknown graph-based model – the Ranking on Data Manifold model, or Manifold Ranking (MR). Particularly, it has been successfullyapplied to content-based image retrieval, because of its outstanding ability to discover underlying geometrical structure of the givenimage database. However, manifold ranking is computationally very expensive, which significantly limits its applicability to largedatabases especially for the cases that the queries are out of the database (new samples). We propose a novel scalable graph-basedranking model called Efficient Manifold Ranking (EMR), trying to address the shortcomings of MR from two main perspectives:scalable graph construction and efficient ranking computation. Specifically, we build an anchor graph on the database instead of atraditional k-nearest neighbor graph, and design a new form of adjacency matrix utilized to speed up the ranking. An approximatemethod is adopted for efficient out-of-sample retrieval. Experimental results on some large scale image databases demonstrate thatEMR is a promising method for real world retrieval applications.Index Terms—Graph-based algorithm, ranking model, image retrieval, out-of-sample1 INTRODUCTIONGRAPH-BASED ranking models have been deeplystudied and widely applied in information retrievalarea. In this paper, we focus on the problem of applyinga novel and efficient graph-based model for contentbasedimage retrieval (CBIR), especially for out-of-sampleretrieval on large scale databases.Traditional image retrieval systems are based on keywordsearch, such as Google and Yahoo image search. Inthese systems, a user keyword (query) is matched withthe context around an image including the title, manualannotation, web document, etc. These systems don’tutilize information from images. However these systemssuffer many problems, such as shortage of the text informationand inconsistency of the meaning of the text andimage. Content-based image retrieval is a considerablechoice to overcome these difficulties. CBIR has drawn agreat attention in the past two decades [1]–[3]. Differentfrom traditional keyword search systems, CBIR systems utilizethe low-level features, including global features (e.g.,color moment, edge histogram, LBP [4]) and local features(e.g., SIFT [5]), automatically extracted from images. A great• B. Xu, J. Bu, C. Chen, and C. Wang are with the Zhejiang ProvincialKey Laboratory of Service Robot, College of Computer Science, ZhejiangUniversity, Hangzhou 310027, China.E-mail: {xbzju, bjj, chenc, wcan}@zju.edu.cn.• D. Cai and X. He are with the State Key Lab of CAD&CG, Collegeof Computer Science, Zhejiang University, Hangzhou 310027, China.E-mail: {dengcai, xiaofeihe}@cad.zju.edu.cn.Manuscript received 9 Oct. 2012; revised 7 Apr. 2013; accepted 22 Apr. 2013.Date of publication 1 May 2013; date of current version 1 Dec. 2014.Recommended for acceptance by H. Zha.For information on obtaining reprints of this article, please send e-mail to:reprints@ieee.org, and reference the Digital Object Identifier below.Digital Object Identifier 10.1109/TKDE.2013.70amount of researches have been performed for designingmore informative low-level features to represent images,or better metrics (e.g., DPF [6]) to measure the perceptualsimilarity, but their performance is restricted by many conditionsand is sensitive to the data. Relevance feedback [7]is a useful tool for interactive CBIR. User’s high level perceptionis captured by dynamically updated weights basedon the user’s feedback.Most traditional methods focus on the data features toomuch but they ignore the underlying structure information,which is of great importance for semantic discovery,especially when the label information is unknown. Manydatabases have underlying cluster or manifold structure.Under such circumstances, the assumption of label consistencyis reasonable [8], [9]. It means that those nearby datapoints, or points belong to the same cluster or manifold,are very likely to share the same semantic label. This phenomenonis extremely important to explore the semanticrelevance when the label information is unknown. In ouropinion, a good CBIR system should consider images’ lowlevelfeatures as well as the intrinsic structure of the imagedatabase.Manifold Ranking (MR) [9], [10], a famous graph-basedranking model, ranks data samples with respect to theintrinsic geometrical structure collectively revealed by alarge number of data. It is exactly in line with our consideration.MR has been widely applied in many applications,and shown to have excellent performance and feasibilityon a variety of data types, such as the text [11], image[12], [13], and video[14]. By taking the underlying structureinto account, manifold ranking assigns each data sample arelative ranking score, instead of an absolute pairwise similarityas traditional ways. The score is treated as a similarity1041-4347 c_ 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.XU ET AL.: EMR: A SCALABLE GRAPH-BASED RANKING MODEL FOR CONTENT-BASED IMAGE RETRIEVAL 103metric defined on the manifold, which is more meaningfulto capturing the semantic relevance degree. He et al. [12]firstly applied MR to CBIR, and significantly improvedimage retrieval performance compared with state-of-the-artalgorithms.However, manifold ranking has its own drawbacks tohandle large scale databases – it has expensive computationalcost, both in graph construction and ranking computationstages. Particularly, it is unknown how to handlean out-of-sample query (a new sample) efficiently underthe existing framework. It is unacceptable to recompute themodel for a new query. That means, original manifold rankingis inadequate for a real world CBIR system, in whichthe user provided query is always an out-of-sample.In this paper, we extend the original manifold rankingand propose a novel framework named Efficient ManifoldRanking (EMR). We try to address the shortcomings ofmanifold ranking from two perspectives: the first is scalablegraph construction; and the second is efficient computation,especially for out-of-sample retrieval. Specifically, webuild an anchor graph on the database instead of the traditionalk-nearest neighbor graph, and design a new form ofadjacency matrix utilized to speed up the ranking computation.The model has two separate stages: an offline stagefor building (or learning) the ranking model and an onlinestage for handling a new query. With EMR, we can handle adatabase with 1 million images and do the online retrievalin a short time. To the best of our knowledge, no previousmanifold ranking based algorithm has run out-of-sampleretrieval on a database in this scale.A preliminary version of this work previously appearedas [13]. In this paper, the new contributions are as follows:• We pay more attention to the out-of-sample retrieval(online stage) and propose an efficient approximatemethod to compute ranking scores for a new queryin Section 4.5. As a result, we can run out-ofsampleretrieval on a large scale database in a shorttime.• We have optimized the EMR code1 and re-run all theexperiments (Section 5). Three new databases includingtwo large scale databases with about 1 millionssamples are added for testing the efficiency of theproposed model. We offer more detailed analysis forexperimental result.• We formally define the formulation of local weightestimation problem (Section 4.1.1) for buildingthe anchor graph and two different methods arecompared to determine which method is better(Section 5.2.2).The rest of this paper is organized as follows. InSection 2, we briefly discuss some related work and inSection 3, we review the algorithm of MR and makean analysis. The proposed approach EMR is described inSection 4. In Section 5, we present the experiment resultson many real world image databases. Finally we provide aconclusions in Section 6.1. http://eagle.zju.edu.cn/∼binxu/2 RELATED WORKThe problem of ranking has recently gained great attentionsin both information retrieval and machine learning areas.Conventional ranking models can be content based models,like the Vector Space Model, BM25, and the language modeling[15]; or link structure based models, like the famousPageRank [16] and HITS [17]; or cross media models [18].Another important category is the learning to rank model,which aims to optimize a ranking function that incorporatesrelevance features and avoids tuning a large numberof parameters empirically [19], [20]. However, many conventionalmodels ignore the important issue of efficiency,which is crucial for a real-time systems, such as a web application.In [21], the authors present a unified framework forjointly optimizing effectiveness and efficiency.In this paper, we focus on a particular kind of rankingmodel – graph-based ranking. It has been successfullyapplied in link-structure analysis of the web [16], [17], [22]–[24], social networks research [25]–[27] and multimedia dataanalysis [28]. Generally, a graph [29] can be denoted asG = (V, E,W), where V is a set of vertices in which eachvertex represents a data point, E ⊆ V × V is a set of edgesconnecting related vertices, and W is a adjacency matrixrecording the pairwise weights between vertices. The objectof a graph-based ranking model is to decide the importanceof a vertex, based on local or global information draw fromthe graph.Agarwal [30] proposed to model the data by a weightedgraph, and incorporated this graph structure into the rankingfunction as a regularizer. Guan et al. [26] proposed agraph-based ranking algorithm for interrelated multi-typeresources to generate personalized tag recommendation.Liu et al. [25] proposed an automatically tag ranking schemeby performing a random walk over a tag similarity graph.In [27], the authors made the music recommendation byranking on a unified hypergraph, combining with richsocial information and music content. Hypergraph is a newgraph-based model and has been studied in many works[31]. Recently, there have been some papers on speeding upmanifold ranking. In [32], the authors partitioned the datainto several parts and computed the ranking function by ablock-wise way.3 MANIFOLD RANKING REVIEWIn this section, we briefly review the manifold ranking algorithmand make a detailed analysis about its drawbacks.Westart form the description of notations.3.1 Notations and FormulationsGiven a set of data χ = {x1, x2, . . . , xn} ⊂ Rm and builda graph on the data (e.g., kNN graph). W ∈ Rn×n denotesthe adjacency matrix with element wij saving the weight ofthe edge between point i and j. Normally the weight canbe defined by the heat kernel wij = exp [ − d2(xi, xj)/2σ2)]if there is an edge linking xi and xj, otherwise wij = 0.Function d(xi, xj) is a distance metric of xi and xj definedon χ, such as the Euclidean distance. Let r:χ → R be aranking function which assigns to each point xi a rankingscore ri. Finally, we define an initial vector y = [y1, . . . , yn]T,in which yi = 1 if xi is a query and yi = 0 otherwise.104 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015The cost function associated with r is defined to beO(r) = 12⎛⎝_ni,j=1wij_ 1 √Diiri − 1 _Djjrj_2 + μ_ni=1_ri − yi_2⎞⎠,(1)where μ > 0 is the regularization parameter and D is adiagonal matrix with Dii =_nj=1 wij.The first term in the cost function is a smoothness constraint,which makes the nearby points in the space havingclose ranking scores. The second term is a fitting constraint,which means the ranking result should fit to theinitial label assignment. With more prior knowledge aboutthe relevance or confidence of each query, we can assigndifferent initial scores to the queries. Minimizing the costfunction respect to r results into the following closed formsolutionr∗ = (In − αS)−1y, (2)where α = 11+μ, In is an identity matrix with n×n, and S isthe symmetrical normalization of W, S = D−1/2WD−1/2. Inlarge scale problems, we prefer to use the iteration scheme:r(t + 1) = αSr(t) + (1 − α)y. (3)During each iteration, each point receives informationfrom its neighbors (first term), and retains its initial information(second term). The iteration process is repeateduntil convergence. When manifold ranking is applied toretrieval (such as image retrieval), after specifying a queryby the user, we can use the closed form or iteration schemeto compute the ranking score of each point. The rankingscore can be viewed as a metric of the manifold distancewhich is more meaningful to measure the semanticrelevance.3.2 AnalysisAlthough manifold ranking has been widely used in manyapplications, it has its own drawbacks to handle large scaledatabased, which significantly limits its applicability.The first is its graph construction method. The kNNgraph is quite appropriate for manifold ranking becauseof its good ability to capture local structure of the data. Butthe construction cost for kNN graph is O(n2 log k), whichis expensive in large scale situations. Moreover, manifoldranking, as well as many other graph-based algorithmsdirectly use the adjacency matrix W in their computation.The storage cost of a sparse W is O(kn). Thus, we need tofind a way to build a graph in both low construction costand small storage space, as well as good ability to captureunderlying structure of the given database.The second, manifold ranking has very expensive computationalcost because of the matrix inversion operationin equation (2). This has been the main bottleneck to applymanifold ranking in large scale applications. Although wecan use the iteration algorithm in equation (3), it is stillinefficient in large scale cases and may arrive at a local convergence.Thus, original manifold ranking is inadequate fora real-time retrieval system.4 EFFICIENT MANIFOLD RANKINGWe address the shortcomings of original MR from twoperspectives: scalable graph construction and efficient rankingcomputation. Particularly, our method can handle theout-of-sample retrieval, which is important for a real-timeretrieval system.4.1 Scalable Graph ConstructionTo handle large databases, we want the graph constructioncost to be sub-linear with the graph size. That means, foreach data point, we can’t search the whole database, as kNNstrategy does. To achieve this requirement, we constructan anchor graph [33], [34] and propose a new design ofadjacency matrix W.The definitions of anchor points and anchor graph haveappeared in some other works. For instance, in [35], theauthors proposed that each data point on the manifoldcan be locally approximated by a linear combination of itsnearby anchor points, and the linear weights become itslocal coordinate coding. Liu et al. [33] designed the adjacencymatrix in a probabilistic measure and used it forscalable semi-supervised learning. This work inspires usmuch.4.1.1 Anchor Graph ConstructionNow we introduce how to use anchor graph to modelthe data [33], [34]. Suppose we have a data set χ ={x1, . . . , xn} ⊂ Rm with n samples in m dimensions, andU = {u1, . . . , ud} ⊂ Rm denotes a set of anchors sharingthe same space with the data set. Let f :χ → R be a realvalue function which assigns each data point in χ a semanticlabel. We aim to find a weight matrix Z ∈ Rd×n thatmeasures the potential relationships between data pointsin χ and anchors in U. Then we estimate f (x) for each datapoint as a weighted average of the labels on anchors       f(xi) =_dk=1zkif (uk), i = 1, . . . , n, (4)with constraints_dk=1 zki = 1 and zki ≥ 0. Element zki representsthe weight between data point xi and anchor uk. Thekey point of the anchor graph construction is how to computethe weight vector zi for each data point xi. Two issuesneed to be considered: (1) the quality of the weight vectorand (2) the cost of the computation.Similar to the idea of LLE [8], a straightforward wayto measure the local weight is to optimize the followingconvex problem:minziε(zi) = 12_xi −_|N(xi)|s=1 us∈N(xi)zis_2s.t._s zis = 1, zi ≥ 0,(5)where N(xi) is the index set of xi’s nearest anchors. Wecall the above problem as the local weight estimation problem.A standard quadratic programming (QP) can solve thisproblem, but QP is very computational expensive. A projectedgradient based algorithm was proposed in [33] tocompute weight matrix and in our previous work [13], akernel regression method was adopted. In this paper, wecompare these two different methods to find the weightvector zi. Both of them are much faster than QP.XU ET AL.: EMR: A SCALABLE GRAPH-BASED RANKING MODEL FOR CONTENT-BASED IMAGE RETRIEVAL 105(1) Solving by Projected GradientThe first method is the projected gradient method, whichhas been used in the work of [33]. The updating rule in thismethod is expressed as the following iterative formula [33]:z(t+1)i= _s(z(t)i− ηt∇ε(zti)), (6)where ηt denotes the step size of time t, ∇ε(z) denotes thegradient of ε at z, and _s(z) denotes the simplex projectionoperator on any z ∈ Rs. Detailed algorithm can be foundin Algorithm 1 of [33].(2) Solving by Kernel RegressionWe adopt the Nadaraya-Watson kernel regression toassign weights smoothly [13]zki =K|xi−uk|λ__dl=1 K|xi−ul|λ_, (7)with the Epanechnikov quadratic kernelKλ(t) =_34(1 − t2) if |t| ≤ 1;0 otherwise.(8)The smoothing parameter λ determines the size of thelocal region in which anchors can affect the target point. Itis reasonable to consider that one data point has the samesemantic label with its nearby anchors in a high probability.There are many ways to determine the parameter λ. Forexample, it can be a constant selected by cross-validationfrom a set of training data. In this paper we use a morerobust way to get λ, which uses the nearest neighborhoodsize s to replace λ, that isλ(xi) = |xi − u[s]|, (9)where u[s] is the sth closest anchor of xi. Later in the experimentpart, we’ll discuss the effectiveness and efficiency ofthe above two methods.Specifically, to build the anchor graph, we connect eachsample to its s nearest anchors and then assign the weights.So the construction has a total complexity O(nd log s), whered is the number of anchors and s is very small. Thus, thenumber of anchors determines the efficiency of the anchorgraph construction. If d  n, the construction is linear tothe database.How can we get the anchors? Active learning [36], [37] orclustering methods are considerable choices. In this paper,we use k-means algorithm and select the centers as anchors.Some fast k-means algorithms [38] can speed up the computation.Random selection is a competitive method which hasextremely low selection cost and acceptable performance.The main feature, also the main advantage of buildingan anchor graph is separating the graph construction intotwo parts – anchor selection and graph construction. Eachdata sample is independent to the other samples but relatedto the anchors only. The construction is always efficientsince it has linear complexity to the date size. Note that wedon’t have to update the anchors frequently, as informativeanchors for a large database are relatively stable (e.g., thecluster centers), even if a few new samples are added.4.1.2 Design of Adjacency MatrixWe present a new approach to design the adjacency matrixW and make an intuitive explanation for it. The weightmatrix Z ∈ Rd×n can be seen as a d dimensional representationof the data X ∈ Rm×n, d is the number of anchorpoints. That is to say, data points can be represented inthe new space, no matter what the original features are.This is a big advantage to handle some high dimensionaldata. Then, with the inner product as the metric to measurethe adjacent weight between data points, we designthe adjacency matrix to be a low-rank form [33], [39]W = ZTZ, (10)which means that if two data points are correlative (Wij >0), they share at least one common anchor point, otherwiseWij = 0. By sharing the same anchors, data pointshave similar semantic concepts in a high probability as ourconsideration. Thus, our design is helpful to explore thesemantic relationships in the data.This formula naturally preserves some good propertiesof W: sparseness and nonnegativeness. The highly sparsematrix Z makes W sparse, which is consistent with theobservation that most of the points in a graph have onlya small amount of edges with other points. The nonnegativeproperty makes the adjacent weight more meaningful:in real world data, the relationship between two items isalways positive or zero, but not negative. Moreover, nonnegativeW guarantees the positive semidefinite property ofthe graph Laplacian in many graph-based algorithms [33].4.2 Efficient Ranking ComputationAfter graph construction, the main computational cost formanifold ranking is the matrix inversion in equation (2),whose complexity is O(n3). So the data size n can not betoo large. Although we can use the iteration algorithm, itis still inefficient for large scale cases.One may argue that the matrix inversion can be done offline,then it is not a problem for on-line search. However,off-line calculation can only handle the case when the queryis already in the graph (an in-sample). If the query is notin the graph (an out-of-sample), for exact graph structure,we have to update the whole graph to add the new queryand compute the matrix inversion in equation (2) again.Thus, the off-line computation doesn’t work for an out-ofsamplequery. Actually, for a real CBIR system, user’s queryis always an out-of-sample.With the form of W = ZTZ , we can rewrite the equation(2), the main step of manifold ranking, by Woodburyformula as follows. Let H = ZD−12 , and S = HTH, then thefinal ranking function r can be directly computed byr∗ = (In − αHTH)−1y =In − HT_HHT − 1αId_−1H_y.(11)By equation (11), the inversion part (taking the mostcomputational cost) changes from a n×n matrix to a d×dmatrix. If d  n, this change can significantly speed upthe calculation of manifold ranking. Thus, applying ourproposed method to a real-time retrieval system is viable,which is a big shortage for original manifold ranking.During the computation process, we never use the adjacencymatrix W. So we don’t save the matrix W in memory,106 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015but save matrix Z instead. In equation (11), D is a diagonalmatrix with Dii =_nj=1 wij. When W = ZTZ,Dii =_nj=1zTizj = zTiv, (12)where zi is the ith column of Z and v =_nj=1 zj. Thus weget the matrix D without using W.A useful trick for computing r∗ in equation (11) is runningit from right to left. So every time we multiply a matrixby a vector, avoiding the matrix – matrix multiplication.As a result, to compute the ranking function, EMR has acomplexity O(dn + d3).4.3 Complexity AnalysisIn this subsection, we make a comprehensive complexityanalysis of MR and EMR, including the computation costand storage cost. As we have mentioned, both MR andEMR have two stages: the graph construction stage andthe ranking computation stage.For the model of MR:• MR builds a kNN graph, i.e., for each data sample,we need to calculate the relationships to its k-nearestneighbors. So the computation cost is O(n2 log k). Atthe same time, we save the adjacency matrix W ∈Rn×n with a storage cost O(kn) since W is sparse.• In the ranking computation stage, the main stepis to compute the matrix inversion in 2, which isapproximately O(n3).For the model of EMR:• EMR builds an anchor graph, i.e., for each data sample,we calculate the relationships to its s-nearestanchors. The computation cost is O(nd log s). We usek-means to select the anchors, we need a cost ofO(Tdn), where T is the iteration number. But thisselection step can be done off-line and unnecessarilyupdated frequently. At the same time, wesave the sparse matrix Z ∈ Rd×n with a storagecost O(sn).• In the ranking computation stage, the main step isEq.(11), which has a computational complexity ofO(dn + d3).As a result, EMR has a computational cost of O(dn) +O(d3) (ignoring s, T) and a storage cost O(sn), while MR hasa computational cost of O(n2) + O(n3) and a storage costO(kn). Obviously, when d  n, EMR has a much lower costthan MR in computation.4.4 EMR for Content-Based Image RetrievalIn this part, we make a brief summary of EMR applied topure content-based image retrieval. To add more information,we just extend the data features.First of all, we extract the low-level features of imagesin the database, and use them as coordinates of data pointsin the graph. We will further discuss the low-level featuresin Section 5. Secondly, we select representative points asanchors and construct the weight matrix Z with a smallneighborhood size s. Anchors are selected off-line and doesFig. 1. Extend matrix W (MR) and Z (EMR) in the gray regions for anout-of-sample.not affect the on-line process. For a stable data set, we don’tfrequently update the anchors. At last, after the user specifyingor uploading an image as a query, we get or extract itslow-level features, update the weight matrix Z, and directlycompute the ranking scores by equation (11). Images withhighest ranking scores are considered as the most relevantand return to the user.4.5 Out-of-Sample RetrievalFor in-sample data retrieval, we can construct the graphand compute the matrix inversion part of equation (2) offline.But for out-of-sample data, the situation is totallydifferent. A big limitation of MR is that, it is hard to handlethe new sample query. A fast strategy for MR is leavingthe original graph unchanged and adding a new row anda new column to W (left picture of Fig. 1). Although thenew W is efficiently to compute, it is not helpful for theranking process (Eq.(2)). Computing Eq.(2) for each newquery in the online stage is unacceptable due to its highcomputational cost.In [40], the authors solve the out-of-sample problemby finding the nearest neighbors of the query and usingthe neighbors as query points. They don’t add the queryinto the graph, therefore their database is static. However,their method may change the query’s initial semantic meaning,and for a large database, the linear search for nearestneighbors is also costly.In contrast, our model EMR can efficiently handle thenew sample as a query for retrieval. In this subsection,we describe the light-weight computation of EMR for anew sample query. We want to emphasize that this is abig improvement over our previous conference version ofthis work, which makes EMR scalable for large-scale imagedatabases (e.g., 1 million samples). We show the algorithmas follows.For one instant retrieval, it is unwise to update the wholegraph or rebuild the anchors, especially on a large database.We believe one point has little effect to the stable anchorsin a large data set (e.g., cluster centers). For EMR, each datapoint (zi) is independently computed, so we assign weightsbetween the new query and its nearby anchors, forming anew column of Z (right picture of Fig. 1).We use zt to denote the new column. Then, Dt = zTtvand ht = ztD−12t , where ht is the new column of H. As wehave described, the main step of EMR is Eq.(11). Our goalis to further speedup the computation of Eq.(11) for a newquery. LetC =_HHT − 1αId_−1=_ni=1hihTi− 1αId_−1, (13)XU ET AL.: EMR: A SCALABLE GRAPH-BASED RANKING MODEL FOR CONTENT-BASED IMAGE RETRIEVAL 107Fig. 2. COREL image samples randomly selected from semantic conceptballoon, beach, and butterfly.and the new C_ with adding the column ht isC_ =_ni=1hihTi+ hthTt− 1αId_−1≈ C (14)when n is large and ht is highly sparse. We can see thematrix C as the inverse of a covariance matrix. The aboveequation says that one single point would not affect thecovariance matrix of a large database. That is to say, thecomputation of C can be done in the off-line stage.The initial query vector yt isyt =_0n1_, (15)where 0n is a n-length zero vector. We can rewrite Eq.(11)with the new query asr(n+1)×1 =_In+1 −_HTChTtC_[H ht]__0n1_. (16)Our focus is the top n elements of r, which is equal torn×1 = −HTCht = Eht. (17)The matrix En×d = −HTC can be computed offline, i.e., inthe online stage, we need to compute a multiplication of an × d matrix and a d × 1 vector only. As ht is sparse (e.g., snon-zero elements), the essential computation is to select scolumns of E according to ht and do a weighted summation.As a result, we need to do sn scalar multiplications and(s − 1)n scalar additions to get the ranking score (rn×1) foreach database sample; while for linear scan using Euclideandistance, we need to do mn scalar subtractions, mn scalarmultiplications and (m−1)n scalar additions. As s  m, ourmodel EMR is much faster than linear scan using Euclideandistance in the online stage.5 EXPERIMENTAL STUDYIn this section, we show several experimental results andcomparisons to evaluate the effectiveness and efficiency ofour proposed method EMR on four real world databases:two middle size databases COREL (5,000 images) andMNIST (70,000 images), and two large size databasesSIFT1M (1 million sift descriptors) and ImageNet (1.2 millionimages). We use COREL and MNIST to compare theranking performance and use SIFT1M and ImageNet toshow the efficiency of EMR for out-of-sample retrieval. OurTABLE 1Statistics of the Four Databasesexperiments are implemented in MATLAB and run on acomputer with 2.0 GHz(×2) CPU, 64GB RAM.5.1 Experiments SetupThe COREL image data set is a subset of COREL imagedatabase consisting of 5,000 images. COREL is widely usedin many CBIR works [2], [41], [42]. All of the images arefrom 50 different categories, with 100 images per category.Images in the same category belong to the same semanticconcept, such as beach, bird, elephant and so on. That isto say, images from the same category are judged relevantand otherwise irrelevant. We use each image as a queryfor testing the in-sample retrieval performance. In Fig. 2,we randomly select and show nine image samples fromthree different categories. In our experiments, we extractfour kinds of effective features for COREL database, includingGrid Color Moment, edge histogram, Gabor WaveletsTexture, Local Binary Pattern and GIST feature. As a result,a 809-dimensional vector is used for each image [43].The MNIST database2 of handwritten digits has a set of70,000 examples. The images were centered in a 28 × 28image by computing the center of mass of the pixels, andtranslating the image so as to position this point at the centerof the 28 × 28 field. We use the first 60,000 images asdatabase images and the rest 10,000 images as queries fortesting the out-of-sample retrieval performance. The normalizedgray-scale values for each pixel are used as imagefeatures.The SIFT1M database contains one million SIFT featuresand each feature is represented by a 128-dimensional vector.The ImageNet is an image database organized accordingto the WordNet nouns hierarchy, in which each node ofthe hierarchy is depicted by hundreds and thousands ofimages3. We downloaded about 1.2 million images’ BoWrepresentations. A visual vocabulary of 1,000 visual wordsis adopted, i.e., each image is represented by a 1,000-lengthvector. Due to the complex structure of the database andhigh diversity of images in each node, as well as the lowquality of simple BoW representation, the retrieval task isvery hard.We use SIFT1M and ImageNet databases to evaluatethe efficiency of EMR on large and high dimensional data.We randomly select 1,000 images as out-of-sample testqueries for each. Some basic statistics of the four databasesare listed in Table 1. For COREL, MNIST and SIFT1Mdatabases, the data samples have dense features, while forImageNet database, the data samples have sparse features.5.1.1 Evaluation Metric DiscussionThere are many measures to evaluate the retrieval resultssuch as precision, recall, F measure, MAP and NDCG [44].2. http://yann.lecun.com/exdb/mnist/3. http://www.image-net.org/index108 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015They are very useful for a real CBIR application, especiallyfor a web application in which only the top returned imagescan attract user interests. Generally, the image retrievalresults are displayed screen by screen. Too many imagesin a screen will confuse the user and drop the experienceevidently. Images in the top pages attract the most interestsand attentions from the user. So the precision at K metricis significant to evaluate the image retrieval performance.MAP (Mean Average Precision) provides a single-figuremeasure of quality across recall levels. MAP has beenshown to have especially good discriminative power andstability. For a single query, Average Precision is the averageof the precision value obtained for the set of top k itemsexisting after each relevant item is retrieved, and this valueis then averaged over all queries [44]. That is, if the set ofrelevant items for a query qj ∈ Q is {d1, . . . , dmj} and Rjk isthe set of ranked retrieval results from the top result untilyou get to item dk, thenMAP(Q) = 1|Q||Q|_j=11mj_mjk=1Precision(Rjk). (18)NDCG is a wildly used metric to evaluate a ranked list[44]. NDCG@K is defined as:NDCG@K = 1IDCG×_Ki=12ri−1log2(i + 1), (19)where ri is 1 if the item at position i is a relevant item and0 otherwise. IDCG is chosen so that the perfect ranking hasa NDCG value 1.5.2 Experiments on COREL DatabaseThe goal of EMR is to improve the speed of manifold rankingwith acceptable ranking accuracy loss. We first compareour model EMR with the original manifold ranking (MR)and fast manifold ranking (FMR [32]) algorithm on CORELdatabase. As both MR and FMR are designed for in-sampleimage retrieval, we use each image as a query and evaluatein-sample retrieval performance. More comparison toranking with SVM can be found in our previous conferenceversion [13]. In this paper, we pay more attention onthe trade-off of accuracy and speed for EMR respect to MR,so we ignore the other methods.We first compare the methods without relevance feedback.Relevance feedback asks users to label some retrievedsamples, making the retrieval procedure inconvenient. Soif possible, we prefer an algorithm having good performancewithout relevance feedback. In Section 5.2.4, weevaluate the performance of the methods after one round ofrelevance feedback. MR-like algorithms can handle the relevancefeedback very efficiently – revising the initial scorevector y.5.2.1 Baseline AlgorithmEud: the baseline method using Euclidean distance forranking.MR: the original manifold ranking algorithm, the mostimportant comparison method. Our goal is to improvethe speed of manifold ranking with acceptable rankingaccuracy loss.TABLE 2Precision and Time Comparisons of TwoWeight Estimation MethodsFMR: fast manifold ranking [32] firstly partitions the datainto several parts (clustering) and computes the matrixinversion by a block-wise way. It uses the SVD techniquewhich is time consuming. So its computational bottleneckis transformed to SVD. When SVD is accurately solved,FMR equals MR. But FMR uses the approximate solution tospeed up the computation. We use 10 clusters and calculatethe approximation of SVD with 10 singular values. Higheraccuracy requires much more computational time.5.2.2 Comparisons of Two Weight Estimation Methodsfor EMRBefore the main experiment of comparing our algorithmEMR to some other models, we use a single experimentto decide which weight estimation method described inSection 4.1.1 should be adopted. We records the averageretrieval precision (each image is used as a query) and thecomputational time (seconds) of EMR with the two weightestimation methods in Table 2.From the table, we see that the two methods havevery close retrieval results. However, the projected gradientis much slower than kernel regression. In the rest ofour experiments, we use the kernel regression method toestimate the local weight (computing Z).5.2.3 PerformanceAn important issue needs to be emphasized: although wehave the image labels (categories), we don’t use them inour algorithm, since in real world applications, labeling isvery expensive. The label information can only be used toevaluation and relevance feedback.Each image is used as a query and the retrieval performanceis averaged. Fig. 3 prints the average precision (at 20to 80) of each method and Table 3 records the average valuesof recall, F1 score, NDCG and MAP (MAP is evaluatedonly for the top-100 returns). For our method EMR, 1000anchors are used. Later in the model selection part, we findthat using 500 anchors achieves a close performance. It iseasy to find that the performance of MR and EMR are veryclose, while FMR lose a little precision due to its approximationby SVD. As EMR’s goal is to improve the speedof manifold ranking with acceptable ranking accuracy loss,the performance results are not to show which method isbetter but to show the ranking performance of EMR is closeto MR on COREL.We also record the offline building time for MR, FMRand EMR in Table 3. For in-sample retrieval, all the threeXU ET AL.: EMR: A SCALABLE GRAPH-BASED RANKING MODEL FOR CONTENT-BASED IMAGE RETRIEVAL 109Fig. 4. Precision at the top 10 returns of the three algorithms on each category of COREL database.methods have the same steps and cost, so we ignore it onCOREL. We find that for a database with 5,000 images, allthe three methods have acceptable building time, and EMRis the most efficient. However, according to the the analysisin Section 4.3, MR’s computational cost is cubic to thedatabase size while EMR is linear to the database size. Theresult can be found in our experiments on MNIST database.The anchor points are computed off-line and do notaffect the current on-line retrieval system. In the workof [13], we have tested different strategies for anchorpoints selection, including normal k-means, fast k-meansand random anchors. The conclusion is that the cost andperformance are trade-offs in many situations.To see the performance distribution in the whole dataset more concretely, we plot the retrieval precision at top10 returns for all 50 categories in Fig. 4. As can be seen, theperformance of each algorithm varies with different categories.We find that EMR is fairly close to MR in almostevery categories, but for FMR, the distribution is totallydifferent.5.2.4 Performance with Relevance FeedbackRelevance Feedback [7] is a powerful interactive techniqueused to improve the performance of image retrieval systems.With user provided relevant/irrelevant informationon the retrieved images, The system can capture the semanticconcept of the query more correctly and graduallyimprove the retrieval precision.Fig. 3. Retrieval precision at top 20 to 80 returns of Eud (left), MR, FMRand EMR (right).Applying relevance feedback to EMR (as well as MR andFMR)is extremely simple.We update the initial vector y andrecompute the ranking scores.We use an automatic labelingstrategy to simulate relevance feedback: for each query, thetop 20 returns’ ground truth labels (relevant or irrelevant tothe query) are used as relevance feedbacks. It is performedfor one round, since the users have no patience to do more.The retrieval performance are plotted in Fig. 5. By relevancefeedback, MR, FMR and EMR get higher retrieval precisionbut still remain close to each other.5.2.5 Model SelectionModel selection plays a key role to many machine learningmethods. In some cases, the performance of an algorithmmay drastically vary by different choices of the parameters,thus we have to estimate the quality of the parameters. Inthis subsection, we evaluate the performance of our methodEMR with different values of the parameters.There are three parameters in our method EMR: s, α,and d. Parameter s is the neighborhood size in the anchorgraph. Small value of s makes the weight matrix Z verysparse. Parameter α is the tradeoff parameter in EMR andMR. Parameter d is the number of anchor points. ForTABLE 3Recall, F1, NCDG and MAP Values, as well as the OfflineBuilding Time (Seconds) of MR, FMR and EMR110 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015Fig. 5. Retrieval precision at top 20 to 80 returns of Eud (left), MR, FMRand EMR (right) after one round of relevance feedback.convenience, the parameter α is fixed at 0.99, consistentwith the experiments performed in [9], [10], [12].Fig. 6 shows the performance of EMR (Precision at 60)by k-means anchors at different values of s. We find thatthe performance of EMR is not sensitive to the selection ofs when s > 3. With small s, we can guarantee the matrix Zhighly sparse, which is helpful to efficient computation. Inour experiments, we just select s = 5.Fig. 7 shows the performance of EMR versus differentnumber of anchors in the whole data set. We findthat the performance increases very slowly when thenumber of anchors is larger than 500 (approximately).In previous experiments, we fix the number of anchorsto 1000. Actually, a smaller number of anchors, like 800or 600 anchors, can achieve a close performance. Withfewer anchors, the graph construction cost will be furtherreduced. But as the size of COREL is not large, the savingis not important.5.3 Experiments on MNIST DatabaseWe also investigate the performance of our method EMR onthe MNIST database. The samples are all gray digit imagesin the size of 28 × 28. We just use the gray values on eachFig. 6. Retrieval precision versus different values of parameter s. Thedotted line represents MR performance.Fig. 7. Retrieval precision versus different number of anchorss. Thedotted line represents MR performance.pixel to represent the images, i.e., for each sample, we usea 784-dimensional vector to represent it. The database wasseparated into 60,000 training data and 10,000 testing data,and the goal is to evaluate the performance on the testingdata. Note that although it is called ’training data’, aretrieval system never uses the given labels. All the rankingmodels use the training data itself to build their modelsand rank the samples according to the queries. Similaridea can be found in many unsupervised hashing algorithms[45], [46] for approximate and fast nearest neighborsearch.With MNIST database, we want to evaluate the efficiencyand effectiveness of the model EMR. As we havementioned, MR’s cost is cubic to the database size, whileEMR is much faster. We record the training time (buildingthe model offline) of MR, FMR and EMR (1k anchors) inTable 4 with the database size increasing step by step. Therequired time for MR and FMR increases very fast and forthe last two sizes, their procedures are out of memory dueto inverse operation. The algorithm MR with the solutionof Eq.(2) is hard to handle the size of MNIST. FMR performseven worse than MR as it clusters the samples andcomputes a large SVD – it seems that FMR is only usefulfor small-size database. However, EMR is much fasterin this test. The time cost scales linearly – 6 seconds for10,000 samples and 35 seconds for 60,000 samples. We usek-means algorithm with maximum 5 iterations to generatethe anchor points. We find that running k-means with 5iterations is good enough for anchor point selection.TABLE 4Computational Time (s) for Offline Training of MR, FMR, andEMR (1k Anchors) on MNIST DatabaseXU ET AL.: EMR: A SCALABLE GRAPH-BASED RANKING MODEL FOR CONTENT-BASED IMAGE RETRIEVAL 111(a) (b) (c)Fig. 8. (a) MAP values with different number of anchors for EMR. (b) Offline training time of EMR with different number of anchors. (c) Online newquery retrieval time of EMR with different number of anchors on MNIST.5.3.1 Out-of-Sample Retrieval TestIn this section, we evaluate the response time of EMRwhen handling an out-of-sample (a new sample). As MR(as well as FMR)’s framework is hard to handle the outof-sample query and is too costly for training the modelon the size of MNIST (Table 4), from now on, we don’tuse MR and FMR as comparisons, but some other rankingscore (similarity or distance) generating methods should becompared. We use the following two methods as baselinemethods:Eud: linear scan by Euclidean distance. This maybe themost simple but meaningful baseline to compare the out-ofsampleretrieval performance. Many previous fast nearestneighbor search algorithms or hashing-based algorithmswere proposed to accelerate the linear scan speed withsome accuracy loss than Euclidean distance. Their goal isdifferent with ranking – the ranking model assigns eachsample a score but not only the neighbors.LSH: locality sensitive hashing [45], a famous hashing codegenerating method. We use LSH to generate binary codesfor the images for both training and testing samples andthen calculate the hamming distance of a query to alldatabase samples as ranking metric. We use 128 bits and256 bits as the code length of LSH.In Fig. 8(a), we draw the MAP (top 200) values for allthe testing data of our model EMR with different numberof anchor points. The performance of Eud and LSHare showed by three horizontal lines. We can see that,when more than 400 anchors are used, EMR outperformsEuclidean distance metric significantly. LSH is worse thanEud due to its binary representation. We also record EMR’soffline training time and online retrieval time in Fig. 8(b)and Fig. 8(c). The computational time for both offline andonline increases linearly to the number of anchors.Then, in Table 5, we record the computational time (inseconds) and out-of-sample retrieval performance of EMR(1000 anchors), Eud and LSH with 128 and 256 code length.The best performance of each line is in bold font. EMR andLSH-128 have close online retrieval time, which is greatlyfaster than linear scan Eud – about 30 times faster. LSHhas very small training cost as its hashing functions arerandomly selected, while EMR needs more time to buildthe model. With more offline building cost, EMR receiveshigher retrieval performance in metric of precision, NDCGat 100 and MAP. The offline cost is valuable. The numberwith ∗ means it is significant higher than Eud at the 0.001significance level.5.3.2 Case StudyFig. 9 is an out-of-sample retrieval case with Fig. 9(a) usingEuclidean distance to measure the similarity and Fig. 9(b)using EMR with 400 anchors and Fig. 9(c) with 600 anchors.Since the database structure is simple, we just need to usea small number of anchors to build our anchor graph.When we use 400 anchors, we have received a good result(Fig. 9(b)). Then, when we use more anchors, we can get abetter result. It is not hard to see that, the results of Fig. 9(b)and (c) are all correct, but the quality of Fig. 9(c) is a littlebetter – the digits are more similar with the query.5.4 Experiments on Large Scale DatabasesIn our consideration, the issue of performance shouldinclude both efficiency and effectiveness. Since our methodis designed to speedup the model ’manifold ranking’, theefficiency is the main point of this paper. The first severalexperiments are used to show that our model is muchfaster than MR in both offline training and online retrievalprocesses, with only a small accuracy loss. The originalMR model can not be directly applied to a large data set,e.g., a data set with 1 million samples. Thus, to show theperformance of our method for large data sets, we comparemany state-of-the-art hash-based fast nearest neighborsearch algorithms (our ranking model can naturally do theTABLE 5Out-of-Sample Retrieval Time (s) and Retrieval PerformanceComparisons of EMR (1k Anchors), Eud and LSH with128 and 256 Code Length on MNIST DatabaseThe best performance is in bold font. The number with ∗ means it is significanthigher than Eud at the 0.001 significance level.112 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015Fig. 9. Top retrieved MNIST digits via (a) Euclidean distance, (b) EMR with 400 anchor points, and (c) EMR with 600 anchor points. The digit in thefirst line is a new query and the rest digits are the top returns.work of nearest neighbor search) on SIFT1M and ImageNetdatabases.For these two sets, there is no exact labels, so we followthe criterion used in many previous fast nearest neighborsearch work [46]: the groundtruth neighbors are obtainedby brute force search. We use the top-1 percent nearestneighbors as groundtruth. We record the computationaltime (offline training and online retrieval) and rankingperformance in Tables 6 and 7. The offline time is for trainingand the online time is for a query retrieval (averaged).We randomly select 1,000 images from the database asout-of-sample queries and evaluate the performance.For comparison, some state-of-the-art hashing methodsincluding LSH, Spectral Hashing [46] and SphericalHashing (a very recent proposed method [47]) are used.For EMR, we select 10% of the database samples to run kmeansalgorithm with maximum 5 iterations,which is veryfast. In the online stage, the hamming distances betweenthe query sample and the database samples are calculatedfor LSH, Spectral hashing and Spherical Hashing and thenthe distances are sorted. While for our method, we directlycompute the scores via Eq.(17) and sort them. If we adoptany filtering strategy to reduce the number of candidatesamples, the computational cost for each method would bereduced equally. So we only compare the largest computationalcost (brute force search). We adopt 64-bit binarycodes for SIFT1M and 128-bit for ImageNet for all the hashmethods.From Tables 6 and 7, we find that EMR has a comparableonline query cost, and a high nearest neighborsearch accuracy, especially on the high dimensional dataset ImageNet, showing its good performance.TABLE 6Computational Time (s) and Retrieval PerformanceComparison of EMR (1k Anchors), and LSH andSpherical Hash on SIFT1M Database(1 Million-Sample, 128-Dim)5.5 Algorithm AnalysisFrom the comprehensive experimental results above, weget a conclusion that our algorithm EMR is effective andefficient. It is appropriate for CBIR since it is friendly tonew queries. A core point of the algorithm is the anchorpoints selection. Two issues should be further discussed: thequality and the number of anchors. Obviously, our goal isto select less anchors with higher quality. We discuss themas follows:• How to select good anchor points? This is an openquestion. In our method, we use k-means clusteringcenters as anchors. So any faster or better clusteringmethods do help to the selection. There is a tradeoffbetween the selection speed and precision. However,the k-means centers are not perfect – some clustersare very close while some clusters are very small.There is still much space for improvement.• How many anchor points we need? There isno standard answer but our experiments providesome clues: SIFT1M and ImageNet databasesare larger than COREL, but they need similarnumber of anchors to receive acceptable results,i.e., the required number of anchors is not proportionalto the database size. This is important,otherwise EMR is less useful. The numberof anchors is determined by the intrinsic clusterstructure.6 CONCLUSIONIn this paper, we propose the Efficient Manifold Rankingalgorithm which extends the original manifold ranking toTABLE 7Computational Time (s) and Retrieval PerformanceComparison of EMR (1k Anchors), and LSHand Spherical Hash on ImageNet Database(1.2 Million-Sample, 1k-Dim)XU ET AL.: EMR: A SCALABLE GRAPH-BASED RANKING MODEL FOR CONTENT-BASED IMAGE RETRIEVAL 113handle large scale databases. EMR tries to address theshortcomings of original manifold ranking from two perspectives:the first is scalable graph construction; and thesecond is efficient computation, especially for out-of-sampleretrieval. Experimental results demonstrate that EMR is feasibleto large scale image retrieval systems – it significantlyreduces the computational time.ACKNOWLEDGMENTSThis work was supported in part by National NaturalScience Foundation of China under Grant 61125203,91120302, 61173186, 61222207, and 61173185, and inpart by the National Basic Research Program of China(973 Program) under Grant 2012CB316400, FundamentalResearch Funds for the Central Universities, Programfor New Century Excellent Talents in University(NCET-09-0685), Zhejiang Provincial Natural ScienceFoundation under Grant Y1101043 and Foundation ofZhejiang Provincial Educational Department under GrantY201018240.

Effective Key Management in DynamicWireless Sensor NetworksAbstract—Recently, wireless sensor networks (WSNs) havebeen deployed for a wide variety of applications, includingmilitary sensing and tracking, patient status monitoring, trafficflow monitoring, where sensory devices often move betweendifferent locations. Securing data and communications requiressuitable encryption key protocols. In this paper, we propose acertificateless-effective key management (CL-EKM) protocol forsecure communication in dynamic WSNs characterized by nodemobility. The CL-EKM supports efficient key updates when anode leaves or joins a cluster and ensures forward and backwardkey secrecy. The protocol also supports efficient key revocationfor compromised nodes and minimizes the impact of a nodecompromise on the security of other communication links.A security analysis of our scheme shows that our protocol is effectivein defending against various attacks.We implement CL-EKMin Contiki OS and simulate it using Cooja simulator to assess itstime, energy, communication, and memory performance.Index Terms—Wireless sensor networks, certificateless publickey cryptography, key management scheme.I. INTRODUCTIONDYNAMIC wireless sensor networks (WSNs), whichenable mobility of sensor nodes, facilitate wider networkcoverage and more accurate service than static WSNs. Therefore,dynamic WSNs are being rapidly adopted in monitoringapplications, such as target tracking in battlefield surveillance,healthcare systems, traffic flow and vehicle status monitoring,dairy cattle health monitoring [9]. However, sensor devicesare vulnerable to malicious attacks such as impersonation,interception, capture or physical destruction, due to theirunattended operative environments and lapses of connectivityin wireless communication [20]. Thus, security is one ofthe most important issues in many critical dynamic WSNapplications. DynamicWSNs thus need to address key securityrequirements, such as node authentication, data confidentialityand integrity, whenever and wherever the nodes move.To address security, encryption key management protocolsfor dynamic WSNs have been proposed in the past basedManuscript received August 6, 2014; revised October 17, 2014; acceptedNovember 18, 2014. Date of publication December 4, 2014; date of currentversion January 13, 2015. This work was supported in part by the BrainKorea 21 Plus Project. The associate editor coordinating the review of thismanuscript and approving it for publication was Prof. Kui Q. Ren.S.-H. Seo is with the Center for Information Security Technologies, KoreaUniversity, Seoul 136-701, Korea (e-mail: seosh77@gmail.com).J. Won, S. Sultana, and E. Bertino are with the Department ofComputer Science, Purdue University, West Lafayette, IN 47907 USA(e-mail: won12@purdue.edu; ssultana@purdue.edu; bertino@purdue.edu).Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TIFS.2014.2375555on symmetric key encryption [1]–[3]. Such type of encryptionis well-suited for sensor nodes because of their limitedenergy and processing capability. However, it suffers from highcommunication overhead and requires large memory space tostore shared pairwise keys. It is also not scalable and notresilient against compromises, and unable to support nodemobility. Therefore symmetric key encryption is not suitablefor dynamic WSNs. More recently, asymmetric key basedapproaches have been proposed for dynamic WSNs [4]–[7],[10], [15], [18], [25], [27]. These approaches take advantageof public key cryptography (PKC) such as elliptic curvecryptography (ECC) or identity-based public key cryptography(ID-PKC) in order to simplify key establishment anddata authentication between nodes. PKC is relatively moreexpensive than symmetric key encryption with respect tocomputational costs. However, recent improvements in theimplementation of ECC [11] have demonstrated the feasibilityof applying PKC to WSNs. For instance, the implementationof 160-bit ECC on an Atmel AT-mega 128, which has an8-bit 8 MHz CPU, shows that an ECC point multiplicationtakes less than one second [11]. Moreover, PKC is moreresilient to node compromise attacks and is more scalableand flexible. However, we found the security weaknessesof existing ECC-based schemes [5], [10], [25] that theseapproaches are vulnerable to message forgery, key compromiseand known-key attacks. Also, we analyzed the critical securityflaws of [15] that the static private key is exposed to the otherwhen both nodes establish the session key. Moreover, theseECC-based schemes with certificates when directly appliedto dynamic WSNs, suffer from the certificate managementoverhead of all the sensor nodes and so are not a practicalapplication for large scale WSNs. The pairing operationbasedID-PKC [4], [18] schemes are inefficient due to thecomputational overhead for pairing operations. To the best ofour knowledge, efficient and secure key management schemesfor dynamic WSNs have not yet been proposed.In this paper, we present a certificateless effective keymanagement (CL-EKM) scheme for dynamic WSNs. In certificatelesspublic key cryptography (CL-PKC) [12], the user’sfull private key is a combination of a partial private keygenerated by a key generation center (KGC) and the user’s ownsecret value. The special organization of the full private/publickey pair removes the need for certificates and also resolves thekey escrow problem by removing the responsibility for theuser’s full private key. We also take the benefit of ECC keysdefined on an additive group with a 160-bit length as secureas the RSA keys with 1024-bit length.1556-6013 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.372 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 10, NO. 2, FEBRUARY 2015In order to dynamically provide both node authenticationand establish a pairwise key between nodes, we buildCL-EKM by utilizing a pairing-free certificateless hybridsigncryption scheme (CL-HSC) proposed by us in an earlierwork [13], [14]. Due to the properties of CL-HSC, thepairwise key of CL-EKM can be efficiently shared betweentwo nodes without requiring taxing pairing operations andthe exchange of certificates. To support node mobility, ourCL-EKM also supports lightweight processes for cluster keyupdates executed when a node moves, and key revocation isexecuted when a node is detected as malicious or leaves thecluster permanently. CL-EKM is scalable in case of additionsof new nodes after network deployment. CL-EKM is secureagainst node compromise, cloning and impersonation, andensures forward and backward secrecy. The security analysisof our scheme shows its effectiveness. Below we summarizethe contributions of this paper:• We show the security weaknesses of existingECC based key management schemes for dynamicWSNs [10], [15], [25].• We propose the first certificateless effective keymanagement scheme (CL-EKM) for dynamic WSNs.CL-EKM supports four types of keys, each of whichis used for a different purpose, including securepair-wise node communication and group-oriented keycommunication within clusters. Efficient key managementprocedures are defined as supporting node movementsacross different clusters and key revocation process forcompromised nodes.• CL-EKM is implemented using Contiki OS and use aTI exp5438 emulator to measure the computation andcommunication overhead of CL-EKM. Also we developa simulator to measure the energy consumption ofCL-EKM. Then, we conduct the simulation of nodemovement by adopting the RandomWalk Mobility Modeland the Manhattan Mobility Model within the grid. Theexperimental results show that our CL-EKM scheme islightweight and hence suitable for dynamic WSNs.The remainder of this paper is organized as follows:In Section 2, we briefly discuss related work and show thesecurity weaknesses of the existing schemes. In Section 3, weprovide our network model and adversary model. In Section 4,we provide an overview of our CL-EKM. In Section 5, weintroduce the details of CL-EKM. In Section 6, we analyzethe security of CL-EKM. In Section 7, we evaluate theperformance of CL-EKM, conduct the simulation of nodemovement in Section 8, and conclude in Section 9.II. RELATED WORKSymmetric key schemes are not viable for mobile sensornodes and thus past approaches have focused only on staticWSNs. A few approaches have been proposed based on PKCto support dynamic WSNs. Thus, in this section, we reviewprevious PKC-based key management schemes for dynamicWSNs and analyze their security weaknesses or disadvantages.Chuang et al. [7] and Agrawal et al. [8] proposed atwo-layered key management scheme and a dynamickey update protocol in dynamic WSNs based on theDiffie-Hellman (DH), respectively. However, bothschemes [7], [8] are not suited for sensors with limitedresources and are unable to perform expensive computationswith large key sizes (e.g. at least 1024 bit). Since ECC iscomputationally more efficient and has a short key length(e.g. 160 bit), several approaches with certificate [5], [10],[15], [25] have been proposed based on ECC. However,since each node must exchange the certificate to establishthe pairwise key and verify each other’s certificate beforeuse, the communication and computation overhead increasedramatically. Also, the BS suffers from the overhead ofcertificate management. Moreover, existing schemes [5], [10],[15], [25] are not secure. Alagheband et al. [5] proposed a keymanagement scheme by using ECC-based signcryption, butthis scheme is insecure against message forgery attacks [16].Huang et al. [15] proposed a ECC-based key establishmentscheme for self-organizing WSNs. However, we found thesecurity weaknesses of their scheme. In step 2 of their scheme,a sensor node U sends z = qU · H(MacKey) + dU (modn)to the other node V for authentication, where qU is astatic private key of U. But, once V receives the z, itcan disclose qU, because V already got MacKey anddU in step 1. So, V can easily obtain qU by computingqU = (z − dU) · H(MacKey)−1. Thus, the sensor node’sprivate key is exposed to the other node during the keyestablishment between two nodes. Zhang et al. [10] proposeda distributed deterministic key management scheme based onECC for dynamic WSNs. It uses the symmetric key approachfor sharing the pairwise key for existing nodes and uses anasymmetric key approach to share the pairwise keys for anew node after deployment. However, since the initial key KIis used to compute the individual keys and the pairwise keysafter deployment for all nodes, if an adversary obtains KI, theadversary has the ability to compute all individual keys andthe pairwise keys for all nodes. Thus, such scheme suffersfrom weak resilience to node compromises. Also, sincesuch scheme uses a simple ECC-based DH key agreementby using each node’s long-term public key and privatekey, the shared pairwise key is static and as a result, isnot secure against known-key attacks and cannot providere-key operation. Du et al. [25] use a ECDSA scheme toverify the identity of a cluster head and a static EC-Diffie-Hellman key agreement scheme to share the pairwise keybetween the cluster heads. Therefore, the scheme by Duet al. is not secure against known-key attacks, because thepairwise key between the cluster heads is static. On the otherhand, Du et al. use a modular arithmetic-based symmetrickey approach to share the pairwise key between a sensornode and a cluster head. Thus, a sensor node cannot directlyestablish a pairwise key with other sensor nodes and, instead,it requires the support of the cluster head. In their scheme, inorder to establish a pairwise key between two nodes in thesame cluster, the cluster head randomly generates a pairwisekey and encrypts it using the shared keys with these twonodes. Then the cluster head transmits the encrypted pairwisekey to each node. Thus, if the cluster head is compromised,the pairwise keys between non-compromised sensor nodesin the same cluster will also be compromised. Therefore,SEO et al.: EFFECTIVE KEY MANAGEMENT IN DYNAMIC WSNs 373Fig. 1. Heterogeneous dynamic wireless sensor network.their scheme is not compromise-resilient against clusterhead capture, because the cluster head randomly generates apairwise key between sensor nodes whenever it is requestedby the nodes. Moreover, in their scheme, in order to share apairwise key between two nodes in different clusters, thesetwo nodes must communicate via their respective clusterheads. So, after one cluster head generates the pairwisekey for two nodes, the cluster head must securely transmitthis key to both its node and the other cluster head. Thus,this pairwise key should be encrypted by using the sharedpairwise key with the other cluster head and the shared keywith its node, respectively. Therefore, if the pairwise keybetween the cluster heads is exposed, all pairwise keys of thetwo nodes in different clusters are disclosed. The scheme byDu et al. supports forward and backward secrecy by using akey update process whenever a new node joins the clusteror if a node is compromised. However, the scheme does notprovide a process to protect against clone and impersonationattack.Most recently, Rahman et al. [4] and Chatterjee et al. [18]have proposed ID-PKC based key management schemessupporting the mobility of nodes in dynamic WSNswhich removes the certificate management overhead.However, their schemes require expensive pairing operations.Although many approaches that enable pairing operations forsensor nodes have been proposed, the computational costrequired for pairing is still considerably higher than standardoperations such as ECC point multiplication. For example,NanoECC, which uses the MIRACL library, takes around17.93s to compute one pairing operation and around 1.27s tocompute one ECC point multiplication on the MICA2(8MHz)mote [17].III. NETWORK AND ADVERSARY MODELSA. Network ModelWe consider a heterogeneous dynamic wireless sensornetwork (See Fig. 1). The network consists of a number ofstationary or mobile sensor nodes and a BS that manages thenetwork and collects data from the sensors. Sensor nodes canbe of two types: (i) nodes with high processing capabilities,referred to as H-sensors, and (ii) nodes with low processingcapabilities, referred to as L-sensors. We assume to haveN nodes in the network with a number N1 of H-sensorsand a number N2 of L-sensors, where N = N1 + N2, andN1 _ N2. Nodes may join and leave the network, and thusthe network size may dynamically change. The H-sensors actas cluster heads while L-sensors act as cluster members. Theyare connected to the BS directly or by a multi-hop path throughother H-sensors. H-sensors and L-sensors can be stationary ormobile. After the network deployment, each H-sensor formsa cluster by discovering the neighboring L-sensors throughbeacon message exchanges. The L-sensors can join a cluster,move to other clusters and also re-join the previous clusters.To maintain the updated list of neighbors and connectivity,the nodes in a cluster periodically exchange very lightweightbeacon messages. The H-sensors report any changes in theirclusters to the BS, for example, when a L-sensor leaves orjoins the cluster. The BS creates a list of legitimate nodes,M, and updates the status of the nodes when an anomalynode or node failure is detected. The BS assigns each nodea unique identifier. A L-sensor nLi is uniquely identified bynode ID Li whereas a H-sensor nHj is assigned a node ID Hj .A Key Generation Center (KGC), hosted at the BS, generatespublic system parameters used for key management by theBS and issues certificateless public/private key pairs for eachnode in the network. In our key management system, a uniqueindividual key, shared only between the node and the BS isassigned to each node. The certificateless public/private keyof a node is used to establish pairwise keys between any twonodes. A cluster key is shared among the nodes in a cluster.B. Adversary Model and Security RequirementsWe assume that the adversary can mount a physical attackon a sensor node after the node is deployed and retrieve secretinformation and data stored in the node. The adversary can alsopopulate the network with the clones of the captured node.Even without capturing a node, an adversary can conduct animpersonation attack by injecting an illegitimate node, whichattempts to impersonate a legitimate node. Adversaries canconduct passive attacks, such as, eavesdropping, replay attack,etc to compromise data confidentiality and integrity. Specificto our proposed key management scheme, the adversary canperform a known-key attack to learn pairwise master keys if itsomehow learns the short-term keys, e.g., pairwise encryptionkeys. As described in [26] and [8], in order to provide a securekey management scheme for WSNs supporting mobile nodes,the following security properties are critical:• Compromise-Resilience: A compromised node must notaffect the security of the keys of other legitimate nodes.In other words, the compromised node must not be ableto reveal pairwise keys of non-compromised nodes. Thecompromise-resilience definition does not mean that anode is resilient against capture attacks or that a capturednode is prevented from sending false data to other nodes,BS, or cluster heads.• Resistance Against Cloning and Impersonation: Thescheme must support node authentication to protectagainst node replication and impersonation attacks.• Forward and Backward Secrecy: The scheme must assureforward secrecy to prevent a node from using an oldkey to continue decrypting new messages. It must alsoassure backward secrecy to prevent a node with the newkey from going backwards in time to decrypt previouslyexchanged messages encrypted with prior keys. forwardand backward secrecy are used to protect against nodecapture attacks.374 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 10, NO. 2, FEBRUARY 2015• Resilience Against Known-Key Attack: The scheme mustbe secure against the known-key attack.IV. OVERVIEW OF THE CERTIFICATELESS EFFECTIVEKEY MANAGEMENT SCHEMEIn this paper, we propose a Certificateless Key Managementscheme (CL-EKM) that supports the establishment of fourtypes of keys, namely: a certificateless public/private key pair,an individual key, a pairwise key, and a cluster key. Thisscheme also utilizes the main algorithms of the CL-HSCscheme [13] in deriving certificateless public/private keys andpairwise keys. We briefly describe the major notations usedin the paper (See Table I), the purpose of these keys and howthey are setup.A. Types of Keys• Certificateless Public/Private Key: Before a node isdeployed, the KGC at the BS generates a uniquecertificateless private/public key pair and installs the keysin the node. This key pair is used to generate a mutuallyauthenticated pairwise key.• Individual Node Key: Each node shares a uniqueindividual key with BS. For example, a L-sensor can usethe individual key to encrypt an alert message sent tothe BS, or if it fails to communicate with the H-sensor.An H-sensor can use its individual key to encrypt themessage corresponding to changes in the cluster. TheBS can also use this key to encrypt any sensitive data,such as compromised node information or commands.Before a node is deployed, the BS assigns the node theindividual key.• Pairwise Key: Each node shares a different pairwise keywith each of its neighboring nodes for secure communicationsand authentication of these nodes. For example, inorder to join a cluster, a L-sensor should share a pairwisekey with the H-sensor. Then, the H-sensor can securelyencrypt and distribute its cluster key to the L-sensorby using the pairwise key. In an aggregation supportiveWSN, the L-sensor can use its pairwise key to securelytransmit the sensed data to the H-sensor. Each nodecan dynamically establish the pairwise key between itselfand another node using their respective certificatelesspublic/private key pairs.• Cluster Key: All nodes in a cluster share a key, named ascluster key. The cluster key is mainly used for securingbroadcast messages in a cluster, e.g., sensitive commandsor the change of member status in a cluster. Only thecluster head can update the cluster key when a L-sensorleaves or joins the cluster.V. THE DETAILS OF CL-EKMThe CL-EKM is comprised of 7 phases: system setup,pairwise key generation, cluster formation, key update, nodemovement, key revocation, and addition of a new node.TABLE ILIST OF NOTATIONSA. System SetupBefore the network deployment, the BS generates systemparameters and registers the node by including it in a memberlist M.1) Generation of System Parameters: The KGC at theBS runs the following steps by taking a security parameterk ∈ Z+ as the input, and returns a list of system parameter_ = {Fq , E/Fq , Gq , P, Ppub = x P, h0, h1, h2, h3} and x.• Choose a k-bit prime q• Determine the tuple {Fq , E/Fq , Gq , P}.• Choose the master private key x ∈R Z∗qand compute thesystem public key Ppub = x P.• Choose cryptographic hash functions {h0, h1,h2, h3} so that h0 : {0, 1}∗ × G2q→ {0, 1}∗, h1 :G3q× {0, 1}∗ × Gq → {0, 1}n, h2 : Gq × {0, 1}∗ ×Gq × {0, 1}∗ × Gq × {0, 1}∗ × Gq → Z∗q, andh3 : Gq×{0, 1}∗×Gq×{0, 1}∗×Gq×{0, 1}∗×Gq → Z∗q.Here, n is the length of a symmetric key.The BS publishes _ and keeps x secret.2) Node Registration: The BS assigns a unique identifier,denoted by Li , to each L-sensor nLi and a unique identifier,denoted by Hj , to each H-sensor nHj, where 1 ≤ i ≤ N1,1 ≤ j ≤ N2, N = N1 + N2. Here we describe the certificatelesspublic/private key and individual node key operationsfor Li , the same mechanisms apply for H-sensors. Duringinitialization, each node nLi chooses a secret value xLi∈R Z∗qand computes PLi= xLi P. Then, the BS requests the KGCfor partial private/public keys of nLi with the input parametersLi and PLi. The KGC chooses rLi∈R Z∗qand then computesa pair of partial public/private key (RLi , dLi ) as below:RLi= rLi PdLi= rLi+ x · h0(Li , RLi , PLi ) mod qThe Li can validate its private key by checking whetherthe condition dLi P = RLi+ h0(Li , RLi , PLi )Ppub holds.SEO et al.: EFFECTIVE KEY MANAGEMENT IN DYNAMIC WSNs 375Li then sets skLi= (dLi , xLi ) as its full private key andpkLi= (PLi , RLi ) as its full public key. The BS also choosesa uniform random number x0 ∈ Z∗qto generate the node’sindividual key K0Li(K0Hjfor nHj ). The individual key iscomputed as an HMAC of x0, Li as followsK0Li= HMAC(x0, Li )After the key generation for all the nodes, the BS generatesa member list M consisting of identifiers and public keysof all these nodes. It also initializes a revocation list R thatenlists the revoked nodes. The public/private key, _, and theindividual key are installed in the memory of each node.B. Pairwise Key GenerationAfter the network deployment, a node may broadcast anadvertisement message to its neighborhood to trigger thepairwise key setup with its neighbors. The advertisementmessage contains its identifier and public key. At first, twonodes set up a long-term pairwise master key between them,which is then used to derive the pairwise encryption key. Thepairwise encryption key is short-term and can be used as asession key to encrypt sensed data.1) Pairwise Master Key Establishment: In this paragraph,we describe the protocol for establishing a pairwise master keybetween any two nodes nA and nB with unique IDs A and B,respectively.We utilize the CL-HSC scheme [13] as a buildingblock. When nA receives an advertisement message from nB,it executes the following encapsulation process to generate along-term pairwise master key KAB and the encapsulated keyinformation, ϕA = (UA,WA).• Choose lA ∈R Z∗qand compute UA = lAP.• ComputeTA = lA · h0(B, RB, PB)Ppub + lA · RB mod qKAB = h1(UA, TA, lA · PB, B, PB)• Computeh = h2(UA, τA, TA, A, PA, B, PB)h_ = h3(UA, τA, TA, A, PA, B, PB)WA = dA + lA · h + xA · h_where τA is a random string to give a freshness.• Output KAB and ϕA = (UA,WA).Then, nA sends A, pkA, τA and ϕA to nB. nB then performsdecapsulation to obtain KAB.• Compute TA = dB · UA.Note: Because of dB = rB + x · h0(B, RB, PB) andUA = lAP mod q, TA is computed as TA = (rB + x ·h0(B, RB, PB)) · lAP mod q = lA · h0(B, RB, PB)Ppub +lA · RB mod q,• Compute h = h2(UA, τA, TA, A, PA, B, PB) andh_ = h3(UA, τA, TA, A, PA, B, PB).• If WA · P = RA +h0(A, RA, PA) · Ppub +h ·UA +h_ · PA,output KAB = h1(UA, TA, xB · UA, B, PB). Otherwise,output invalid.TABLE IICLUSTER FORMATION PROCESS2) Pairwise Encryption Key Establishment: OncenA and nB set the pairwise master key KAB, they generatean HMAC of KAB and a nonce r ∈R Z∗q. The HMAC is thenvalidated by both nA and nB. If the validation is successful,the HMAC value is established as the short-term pairwiseencryption key kAB. The process is summarized below:• nB chooses a random nonce r ∈R Z∗q, computeskAB = HMAC(KAB, r ) and C1 = EkAB (r, A, B). Then,nB sends r and C1 to nA.• When nA receives r and C1, it computeskAB = HMAC(KAB, r ) and decrypts C1. Then itvalidates r , A and B and if valid confirms that nB knowsKAB and it can compute kAB.C. Cluster FormationOnce the nodes are deployed, each H-sensor discoversneighboring L-sensors through beacon message exchanges andthen proceeds to authenticate them. If the authentication issuccessful, the H-sensor forms a cluster with the authenticatedL-sensors and they share a common cluster key. TheH-sensor also establishes a pairwise key with each memberof the cluster. To simplify the discussion, we focus on theoperations within one cluster and consider the j th cluster.We also assume that the cluster head H-sensor is nHj withnLi (1 ≤ i ≤ n) as cluster members. nHj establishes a clusterkey GKj for secure communication in the cluster. Table IIshows the cluster formation process.1) Node Discovery and Authentication: For node discovery,nHj broadcasts an advertisement message containingHj and pkHj. Once nLi within Hj ’s radio range receivesthe advertisement, it checks Hj and pkHj , and initiatesthe Pairwise Key Generation procedure. Note that nLi mayreceive multiple advertisement messages if it is within therange of more than one H-sensor. However, nLi must chooseone H-sensor, may be by prioritizing over the proximity andsignal strength. Additionally, nLi can record other H-sensoradvertisements as backup cluster heads in the event that theprimary cluster head is disabled. If nLi selects multiple clusterheads and sends a response to all of them, it is considered as376 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 10, NO. 2, FEBRUARY 2015a compromised node. nLi and nHj perform the Pairwise KeyGeneration procedure to obtain a pairwise master key, KLi Hjand a pairwise encryption key, kLi Hj .2) Cluster Key Generation: nHj chooses x j ∈R Z∗qtogenerate a cluster key GKj as followsGKj = HMAC(x j , Hj )Then, nHj computes C2 = EkLi Hj(GKj , Hj , Li ) to distributethe GKj. Then nHj sends Hj and C2 to nLi . nLi decryptsC2 to recover Hj , Li and GKj by using kLi Hj . If nLi fails tocheck Hj , Li , it discards the message and reports nHj to theBS as an illegitimate cluster head. Otherwise, nLi confirms thatnHj is valid and can compute GKj . Then, nLi stores GKj asa cluster key. Next, nLi computes HMAC(kLiHj ,GKj ) andC3 = EkLi Hj(Li ,HMAC(kLi Hj ,GKj )). It transmits C3 andLi to nHj. After nHj receives messages from nLi , it decryptsC3 by using kLi Hj . Then it checks Li and the validity ofHMAC(kLiHj ,GKj ). If the validity check fails, nHj discardsthe message. Otherwise, nHj can confirm that nLi shares thevalid GKj and kHj Li . nHj adds Li and pkLi on member listof the j th cluster, Mj .3) Membership Validation: After discovering all the neighboringnodes nLi (1 ≤ i ≤ n) in the j th cluster, nHj computesC4 = EK0Hj(Hj ,Mj ) and transmits C4 and Hj to the BS.After receiving messages from nHj , the BS checks the validityof the nodes listed in Mj . If all nodes are legitimate, the BSsends an acknowledgement to nHj . Otherwise, the BS rejectsMj and investigates the identities of invalid nodes (false orduplicate ID). Then, the BS adds the identities of invalid nodesto the revocation list and reports it to nHj . Upon receiving theacknowledge message, nHj computes C5 = EGKj (Hj ,Mj )and broadcasts C5 to all the nodes in j th cluster.D. Key UpdateIn order to protect against cryptanalysis and mitigatedamage from compromised keys, frequent encryption keyupdates are commonly required. In this section we providethe pairwise key update and cluster key update operations.1) Pairwise Key Update: To update a pairwise encryptionkey, two nodes which shared the pairwise key perform aPairwise Encryption Key Establishment process. On the otherhand, the pairwise master key does not require periodicalupdates, because it is not directly used to encrypt each sessionmessage. As long as the nodes are not compromised, the pairwisemaster keys cannot be exposed. However, if a pairwisemaster key is modified or needs to be updated according tothe policy of the BS, the Pairwise Master Key Establishmentprocess must be executed.2) Cluster Key Update: Only cluster head H-sensors canupdate their cluster key. If a L-sensor attempts to changethe cluster key, the node is considered a malicious node.The operation for any j th cluster is described as follows:1) nHj chooses x_j∈R Z∗qand computes a new cluster keyGK_j= HMAC(x_j , Hj ). nHj also generates an Updatemessage including HMAC(GK_j ,Update) and computesC6 = EGKj (GK_j ,HMAC(GK_j ,Update)). Then, nHjtransmits Update and C6 to its cluster members.2) Each member nLi decrypts C6 using the GKj , verifiesHMAC(GK_j ,Update) and updates a cluster key as GK_j .Then, each nLi sends the acknowledgement message to nHj .E. Node MovementWhen a node moves between clusters, the H-sensorsmust properly manage the cluster keys to ensure theforward/backward secrecy. Thus, the H-sensor updates thecluster key and notifies the BS of the changed node status.Through this report, the BS can immediately update the nodestatus in the M. We denote a moving node as nLm .1) Node Leave: A node may leave a cluster due to nodefailure, location change or intermittent communication failure.There are both proactive and reactive ways for the cluster headto detect when a node leaves the cluster. The proactive caseoccurs when the node nLm actively decides to leave the clusterand notifies the cluster head nHj or the cluster head decidesto revoke the node. Since in this case nHj can confirm that thenode has left, it transmits a report EK0Hj(NodeLeave, Lm) toinform the BS that nLm has left the cluster. After receivingthe report, the BS updates the status of nLm in M and sendsan acknowledgement to nHj . The reactive case occurs whenthe cluster head nHj fails to communicate with nLm. It mayhappen that a node dies out of battery power, fails to connectto nHj due to interference or obstacles, is captured by theattacker or is moved unintentionally. Since the nodes in acluster periodically exchange lightweight beacon messages,nHj can detect a disappeared node nLm when it does notreceive the beacon message from nLm for a predeterminedtime period. So, nHj reports the status of the node nLmto the BS by sending EK0Hj(NodeDisappear, Lm). Whenthe BS receives the report, it updates the status of nLm inthe M and acknowledges to nHj. Once nHj receives theacknowledgement from the BS, it changes its cluster keywith the following operations: 1) nHj chooses a new clusterkey GK_j and computes EkLi Hj(GK_j , NodeLeave, Lm) usingpairwise session keys with each node in its cluster, except nLm .2) Then, nHj sends EkLi Hj(GK_j , NodeLeave, Lm) to eachmember node except nLm . 3) Each nLi decrypts it using kLi Hjand updates the cluster key as GK_j .2) Node Join: Once the moving node nLm leaves a cluster,it may join other clusters or return to the previous cluster aftersome period. For the sake of simplicity, we assume that nLmwants to join the lth cluster or return to the j th cluster.(i) Join a New Cluster: nLm sends a join request whichcontains Ln+1 and pkLn+1 to join a lth cluster. After nHlreceives the join request, nLm and nHl perform PairwiseKey Generation procedure to generate KLm Hl and kLm Hl ,respectively. Next, nHl transmits EK0Hl(NodeJoin, Lm)to the BS. The BS decrypts the message and validateswhether nLm is a legitimate node or not and sends anacknowledgement to nHl if successful. The BS alsoupdates the node member list, M. In case of nodevalidation failure at the BS, nHl stops this processand revokes the pairwise key with nLm. Once nHlSEO et al.: EFFECTIVE KEY MANAGEMENT IN DYNAMIC WSNs 377receives the acknowledgement, it performs the ClusterKey Update process with all other nodes in the cluster.nHl also computes EkLm Hl(GK_l , Hl , Lm), and sends itto the newly joined node nLm .(ii) Return to the Previous Cluster: nLm sends a join requestwhich contains Ln+1 and pkLn+1 to join a j th cluster.Once nHj receives the join request, it checks a timerfor nLm which is initially set to the Thold . Thold indicatesthe waiting time before discarding the pairwise masterkey when a L-sensor leaves. If nLm returns to the j thcluster before the timer expires, nLm and nHj performonly the Pairwise Encryption Key Establishment procedureto create a new pairwise encryption key, k_LmHj.Otherwise, they perform the Pairwise Key Generationprocedure to generate a new K_LmHland k_LmHl, respectively.Then, the cluster head nHj also updates the clusterkey to protect backward key secrecy. Before updatingthe cluster key, nHj transmits EK0Hj(NodeReJoin, Lm)to the BS. Once the BS decrypts the message anddetermines that nLm is a valid node, the BS sends theacknowledgement to nHl . The BS then updates the memberlist M. Once nHl receives the acknowledgement,it performs the Cluster Key Update process with allother nodes in the cluster. Afterwards, nHj computesEk_Lm Hj(GK_j , Hj , Lm) and sends it to nLm .F. Key RevocationWe assume that the BS can detect compromisedL-sensors and H-sensors. The BS may have an intrusiondetection system or mechanism to detect malicious nodes oradversaries [19], [20]. Although we do not cover how the BScan discover a compromised node or cluster head in this paper,the BS can utilize the updated node status information of eachcluster to investigate an abnormal node. In our protocol, acluster head reports the change of its node status to the BS,such as whenever a node joins or leaves a cluster. Thus, the BScan promptly manage the node status in the member list, M.For instance, the BS can consider a node as compromisedif the node disappears for a certain period of time. In thatcase, the BS must investigate the suspicious node and itcan utilize the node fault detection mechanism introducedin [21] and [22]. In this procedure, we provide a key revocationprocess to be used when the BS discovers a compromised nodeor a compromised cluster head. We denote a compromisednode by nLc in the j th cluster for a compromise node caseand a compromised head by nHj for a compromise clusterhead case.1) Compromised Node: The BS generates a CompNodemessage and a EK0Hj(CompNode, Lc). Then it sendsEK0Hj(CompNode, Lc) to all nHj , (1 ≤ j ≤ N2). After allH-sensors decrypt the message, they update the revocationlist of their clusters. Then, if related keys with nLc exist, therelated keys are discarded. Other than nLc , nHj performs theNode leave operations to change the current cluster key withthe remaining member nodes.2) Compromised Cluster Head: After the BS generates aCompHeader message and a EK0Li(CompHeader, Hj), itsends the message to all nLi (1 ≤ i ≤ n) in the j th cluster. TheBS also computes EK0Hi(CompHeader, Hj), (1 ≤ i ≤ N2,i _= j ) and transmits it to all H-sensors except nHj. Onceall nodes decrypt the message, they discard the related keyswith nHj . Then, each nLi attempts to find other neighboringcluster heads and performs the Join other cluster steps of theNode join process with the neighboring cluster head. If somenode nLi is unable to find another cluster head node, it mustnotify the BS by sending EK0Li(FindNewClusteLi ). The BSproceeds to find the nearest cluster head nHn for nLi andconnect nHn with nLi . Then, they can perform the Join othercluster steps.G. Addition of a New NodeBefore adding a new node into an existing networks, theBS must ensure that the node is not compromised. Thenew node nLn+1 establishes a full private/public key throughthe node registration phase. Then, the public systemparameters, a full private/public key and individualkey K0Ln+1are stored into nLn+1 . The BS generatesEK0Hj(NewNode, Ln+1, pkLn+1) and sends it to all nHj ,(1 ≤ j ≤ N2). After nLn+1 is deployed in the network,it broadcasts an advertisement message which containsLn+1 and pkLn+1 to join a neighboring cluster. If multipleH-sensors receive nLn+1’s message, they will transmit aResponse message to nLn+1 . nLn+1 must choose one H-sensorfor a valid registration. If nLn+1 selects nHj according to thedistance and the strength of signal, it initiates the PairwiseKey Generation procedure. In order to provide backwardsecrecy, nHj performs Cluster Key Update procedure, wherethe Update message contains Ln+1 and pkLn+1. Then, nHjcomputes C7 = EkLn+1 Hj(GK_j , Hj , Ln+1), and sends C7and Hj to nLn+1. After nLn+1’s registration, nHj transmitsEK0Hj(NodeJoin, Ln+1) to the BS. Once the BS decrypts themessage, it updates the status of the node nLn+1 in memberlist, M.VI. SECURITY ANALYSISFirst, we briefly discuss the security of CL-HSC [13]which is utilized as a building block of CL-EKM. Later,we discuss how CL-EKM achieves our security goals. TheCL-HSC [13] provides both confidentiality and unforgeabilityfor signcrypted messages based on the intractability of theEC-CDH1 Moreover, it is not possible to forge or expose thefull private key of an entity based on the difficulty of EC-CDH,without the knowledge of both KGC’s master private key andan entity’s secret value. Here, the confidentiality is definedas indistinguishability against adaptive chosen ciphertext andidentity attacks (IND-CCA2) while unforgeability is defined1The Elliptic Curve Computational Diffie-Hellman problem (EC-CDH) isdefined as follows: Given a random instance (P,aP, bP) ∈ Gq for a,b ∈R Z∗q, compute abP.378 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 10, NO. 2, FEBRUARY 2015as existential unforgeability against adaptive chosen messagesand identity attacks (EUF-CMA). Further details on theCL-HSC scheme and its security proof are provided in [13].A. Compromise-Resilience of CL-EKMWe assume that an adversary captures a node nLi in thej th cluster. This adversary can then extract the keys of nLi ,such as the pairwise key shared with the cluster head nHj ,the public/private key pair, the cluster key GKj, and theindividual key. However, the pairwise master/encryption keygeneration between any two nodes are independent of others,and hence each pair of nodes has different pairwise keys.Therefore, even if the adversary manages to obtain nLi’s keys,it is unable to extract any information useful to compromisethe pairwise keys of other uncompromised nodes. Moreover,due to the intractability of EC-CDH problem, the adversarycannot obtain the KGC’s master private key x from nLi’spublic/private keys pkLi /skLi . As a result, the compromiseof a sensor does not affect the communication security amongother L-sensors or H-sensors. Even though the attacker canread the group communications within the cluster with thecluster key extracted from the compromised node, it cannotget any information about the cluster key of other clusters.B. Resistance Against Cloning and Impersonation AttackAn adversary can conduct the cloning attack if a node iscaptured; the key is then extracted and the node is replicatedin another neighborhood. However, since the cluster headvalidates each node with the BS in the node join process ofour CL-EKM, the BS is able to detect a cloned node when itis placed in an unintended cluster. After the BS investigatesthe cloned node, it revokes the node and notifies the noderevocation to all cluster heads. Thus, although the cloned nodemay try to join other clusters, the cluster head will abort eachattempt. Therefore, our scheme is resistant against the cloningattack.The adversary may also attempt an impersonation attackby inserting an illegitimate node nC. Assume that a node nCposes as nLi . The node ID Li and public key, pkLi=(PLi , RLi ) are publicly known within the network. Hence,nC can broadcast Li and pkLi. When nL j receives themessage, it will compute the pairwise master key KLi L j ,and the encapsulated key information ϕL j= (UL j ,WL j )towards establishing the pairwise Master key. As the next step,nL j sends     ϕL j , L j , pkL j to nC for decapsulation, whichrequires nC to compute TL j as (dLi· UL j ). However, nCfails to compute TL j since nC has no knowledge of nLi’spartial private key dLi . Moreover due to the intractability ofEC-CDH1, the adversary cannot forge dLi without the knowledgeof the KGC’s master private key. Thus, nC is unableto generate a legitimate pairwise master key, KLi L j. However,nC may try to establish the pairwise encryption with a randomkey K_, rather than generating a legitimate master key. To thisend, nC chooses a random nonce r , computes an encryptionkey k_ as HMAC(r, K_) and sends          r, E_k (r, Li , L j ) to nL j .However, nC cannot successfully pass the validation at nL j ,since nL j first computes the pairwise encryption key withnL j as kLi L j= HMAC(r, KLi L j ) and then tries to decryptE_k (r, Li , L j ) using kLi L j . Thus, nL j fails to decrypt andhence, it does not confirm the pairwise encryption key to nC,which is then reported to the BS. Thus, CL-EKM is resistantagainst impersonation attacks.C. Forward and Backward SecrecyIn CL-EKM, messages exchanged between nodes or withina cluster are encrypted with the pairwise encryption key orcluster key. CL-EKM provides the key update and revocationprocesses to ensure forward secrecy when a node leaves orcompromised node is detected. Using key update process,CL-EKM ensures backward secrecy when a new node joins.Once a node is revoked from the network, all its keys areinvalidated and the associated cluster key is updated. Thecluster head sends the new cluster key to each cluster node,except the revoked node, by encrypting the key with thepairwise encryption key between the cluster and each intendednode. Thus, the revoked node fails to decrypt any subsequentmessages using the old pairwise encryption key or cluster key.When a node joins a cluster, the cluster head generates a newcluster key by choosing a new random value. Since the joinednode receives the new cluster key, it cannot decrypt earliermessages encrypted using the older cluster keys.D. Resistance Against Known-Key AttackWe assume that an adversary obtains the current pairwiseencryption key kLi Hj= HMAC(KLi Hj , r ) betweennLi and nHj and conducts the known-key attack. The adversarymay attempt to extract the long term pairwise master keyKLi Hj using kLi Hj . However, due to the one-way featureof HMAC(.), the adversary fails to learn KLi Hj. Also,when nLi and nHj update the pairwise encryption key ask_Li Hj= HMAC(KLi Hj , r _), the adversary cannot computethe updated pairwise encryption key k_Li Hj, without the knowledgeof KLi Hj . Thus, CL-EKM is resistant against known-keyattack when the pairwise encryption key is compromised.VII. PERFORMANCE EVALUATIONWe implemented CL-EKM in Contiki OS [29] andused Contiki port [28] of TinyECC [24] for elliptic curvecryptography library. In order to evaluate our scheme, weuse the Contiki simulator COOJA. We run emulations on thestate-of-the-art sensor platform TI EXP5438 which has 16-bitCPU MSP430F5438A with 256KB flash and 16KB RAM.MSP430F5438A has 25MHz clock frequency and can belowered for power saving.A. Performance Analysis of CL-EKMWe measure the individual performance of the three stepsin the pairwise master/encryption key establishment process,namely, (i) encapsulation, (ii) decapsulation, and (iii) pairwiseencryption key generation. We evaluate each step in termsof (i) computation time, and (ii) energy consumption.In this experiment, we vary the processing power i.e. CPUclock rate of the sensors since we consider heterogeneousSEO et al.: EFFECTIVE KEY MANAGEMENT IN DYNAMIC WSNs 379Fig. 2. Computation overhead for pairwise master/encryption key establishment. (a) Encapsulating key information. (b) Decapsulating key information.(c) Pairwise encryption key establishment.Fig. 3. Energy consumption for pairwise master/encryption key establishment. (a) Encapsulating key information. (b) Decapsulating key information.(c) Pairwise encryption key establishment.WSNs with H-sensors being more powerful. Three differentelliptic curves recommended by SECG (Standards for EfficientCryptography Group) [30], i.e., (i) secp128r2 (128-bit ECC),(ii) secp160r1 (160-bit ECC), and (iii) secp192r1 (192-bitECC), are used for the experiment.Fig. 2 shows the time for the pairwise key generationprocess. As expected, the pairwise master key generationtakes most of the time due to the ECC operations(See Fig. 2(a), 2(b)). However, it is important to mention thatthe pairwise master key is used only to derive the short-termpairwise encryption key. Once two nodes establish the pairwisekeys, they do not require further ECC operations. Fig. 2(a)shows the computation times of the encapsulation process forvarious CPU clock rates of the sensor device. The computationtime increases with the ECC key bit length. secp192r1 needsalmost 1.5 times more time than secp160r1. secp128r2 takesapproximately 4% less time than secp160r1. If CPU clockrate is set to 25MHz and secp160r1 is adopted, 5.7 secondsare needed for encapsulation of key. Fig. 2(b) shows theprocessing time for the decapsulation. Decapsulation requiresabout 1.57 times more CPU computation time than encapsulation.This is because decapsulation has six ECC pointmultiplications, whereas encapsulation includes only four ECCpoint multiplications. Finally, the computation time for pairwiseencryption key establishment is shown in Fig. 2(c).At 25MHz CPU clock rate, it requires 5 ms, which is negligiblecompared to the first two steps. This is due to the factthat this step just needs one HMAC and one 128-bit AESoperation. Next, we measure the energy consumption. As wecan see from Fig. 3, the faster the processing power (i.e. CPUclock rate) is, the more energy is consumed. However, asshown in Fig. 3(a) and Fig. 3(b), there is no differencebetween 16MHz and 25MHz while 25MHz results in fastercomputation than 16MHz. In addition, secp160r1 might bea good choice for elliptic curve selection, since it is moresecure than secp128r2 and consumes reasonable CPU timeand energy for WSNs. In our subsequent experiments, weutilize secp160r1.B. Performance ComparisonsIn this section, we benchmark our scheme with three previousECC-based key management schemes for dynamic WSNs:HKEP [15], MAKM [25] and EDDK [10]. Due to the variabilityof every schemes, we chose to compare a performance ofthe pairwise master key generation step because it is the mosttime consuming portion in each of the schemes. We measuredthe total energy consumption of computation and communicationto establish a pairwise key between two L-sensors. Forthe experiment, we implemented four schemes on TI EXP5438at 25MHz using ECC with secp160r1 parameters andAES-128 symmetric key encryption. EC point is compressedto reduce the packet size and LPL (Low Power Listening)is utilized for power conservation. Thus, sensors wake upfor short durations to check for transmissions every second.If no transmission is detected, they revert to a sleep mode.To compute the energy consumption for communication, weutilize the energy consumption data of CC2420 from [27]and IEEE 802.15.4 protocol overhead data from [31]. We considertwo scenarios as shown in Fig. 4. In the first scenario,two L-sensors lie within a 1-hop range, but the distancebetween the H-sensor and the L-sensor varies from 1 to 8(see Fig. 4 (a)). In the second scenario, two L-sensors andthe H-sensor lie in a 1-hop range, but the wireless channelconditions are changed (see Fig. 4 (b)). When a wirelesschannel condition is poor, a sender may attempt to resend apacket to a destination multiple times. Expected TransmissionCount (ETX) is the expected number of packet transmissionto be received at the destination without error. Fig. 5shows the energy consumption of the four schemes for apairwise key establishment when the number of hops betweenL-sensors and H-sensor, n, increases. When n is one, HKEP380 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 10, NO. 2, FEBRUARY 2015Fig. 4. Network topology.Fig. 5. Energy consumption comparison for pairwise key establishment inscenario (a).requires more energy than our scheme because it performs sixmessage exchanges to establish a pairwise key between twoL-sensors, while our scheme needs just two messageexchanges. When n is one, MAKM consumes the leastenergy because the L-sensor performs a single AESsymmetric encryption, but other schemes run expensive ECCoperations. However, as n increases, the energy of MAKMincreases because the H-sensor is always involved in thegeneration of a pairwise key between two L-sensors. As aresult, MAKM consumes more energy than our scheme whenn is larger than one and the gap also widens when n increases.A packet delivery in a wireless sensor network is unreliabledue to unexpected obstacles, time-varying wireless channelconditions, and a low-power transceiver. Fig. 6 shows theenergy consumption of the four schemes for a pairwisekey establishment when ETX varies from 1 to 4. As ETXincreases, the energy consumption of HKEP increases morerapidly, because it requires six message exchanges. Also,HKEP is insecure, because the static private key of a nodeis exposed to the other node while the two nodes establishthe session key. Although EDDK and MAKM may showbetter performance due to low computational overhead, thedifference between MAKM and our scheme is only 0.121 Jand the difference between EDDK and our scheme is 0.045 J.Both EDDK and MAKM are insecure against the knownkeyattack and do not provide a re-keying operation for thecompromised pairwise key. EDDK also suffers from weakresilience to node compromises. Therefore, this performanceevaluation demonstrates that overall, our scheme outperformsthe existing schemes in terms of a better trade-off between thedesired security properties and energy consumption includingcomputational and communication overhead.VIII. SIMULATION OF NODE MOVEMENTSA. SettingWe developed a simulator which counts the keymanagement-related events and yields total energy consumptionfor key-management-related computations using the datain Sec. 7.1. We focus on the effects of node movement andFig. 6. Energy consumption comparison for pairwise key establishment inscenario (b).Fig. 7. Network topology for simulation.do not consider the impact of lower network layer protocols.We consider a 400×400 m2 space with 25 H-sensors placedon the grid corners (see Fig. 7). In CL-EKM, an H-sensormaintains two timers: Tbackof f and Thold to efficiently managethe cluster when a node moves. Tbackof f denotes the clusterkey update frequency. If Tbackof f = 0, the cluster key isupdated whenever a node joins or leaves. Otherwise, theH-sensor waits a time equal to Tbackof f after a node joinsor leaves to update the cluster key. Thold denotes the waitingtime before discarding the pairwise master key when aL-sensor leaves. If Thold = 0, the pairwise master key witha L-sensor is revoked right after the node leaves the cluster.Otherwise, the H-sensor stores the pairwise master key withthe left L-sensor for a time equal to Thold . For the movementof L-sensor, we adopt two well-known mobility models usedfor simulation of mobile ad-hoc network: the Random WalkMobility Model and the Manhattan Mobility Model [32].H-sensors are set to be stationary since they are usually partof the static infrastructure in real world applications.1) Random Walk Mobility Model: The Random WalkMobility Model mimics the unpredictable movements of manyobjects in nature. In our simulation, 1,000 L-sensors arerandomly distributed. Each L-sensor randomly selects anH-sensor among the four H-sensors in its vicinity and establishesthe pairwise key and cluster key. After the simulationstarts, the L-sensors randomly select a direction and moveat a random speed uniformly selected from [0, 2VL] (i.e., themean speed = VL ). The new direction and speed are randomlyselected every second. If a L-sensor crosses a line, it firstchecks whether it is still connected with its current H-sensor.If not, the node attempts to find an H-sensor which it hadSEO et al.: EFFECTIVE KEY MANAGEMENT IN DYNAMIC WSNs 381Fig. 8. Node movement simulation results in random walk mobility model. (a) Energy consumption of one H-sensor for cluster key update for one day(Thold = 100 sec). (b) Energy consumption of one H-sensor for pairwise key establishment for one day (Tbacko f f = 6 sec). (c) Energy consumption of oneL-sensor for pairwise key establishment for one day (Tbacko f f = 6 sec).Fig. 9. Node movement simulation results in Manhattan mobility model. (a) Energy consumption of one H-sensor for cluster key update for one day(Thold = 100 sec). (b) Energy consumption of one H-sensor for pairwise key establishment for one day (Tbacko f f = 6 sec). (c) Energy consumption of oneL-sensor for pairwise key establishment for one day (Tbacko f f = 6 sec).previously connected to and it still maintains a pairwise masterkey. In the case of a failure, the node randomly selects anH-sensor among the surrounding H-sensors.2) Manhattan Mobility Model: The Manhattan MobilityModel mimics the movement patterns in an urban area organizedaccording to streets and roads. In our simulation, 1,000L-sensors are randomly distributed and move in a grid. Theycan communicate with two adjacent H-sensors. Each L-sensorrandomly selects its direction and chooses an H-sensor withinits path as its cluster head. After the simulation starts, theL-sensors move at a random speed uniformly selected from[0, 2VL]. At each intersection, a L-sensor has 0.5 probabilityof moving straight and a 0.25 probability of turning left orright. If a L-sensor arrives at a new intersection, it first choosesa new direction and checks whether it is still connected withits current H-sensor. If not, it chooses an H-sensor within itsnew vector as its new cluster head.B. The Effect of Tbackof fFig. 8(a) shows the energy consumption of an H-sensorfor a cluster key update during the course of a day ina Random Walk Model. As Tbackof f increases, the energyconsumption decreases since the number of cluster key updatesis reduced. The faster VL is, the more rapidly the energyconsumption decreases as Tbackof f increases since L-sensorsfrequently cross the border lines. This tendency also showsin the Manhattan Mobility Model (see Fig. 9(a)). However,the H-sensors consume more energy at low speeds than inthe Random Walk Mobility Model since the L-sensors do notchange directions until they reach an intersection. A largerTbackof f means a lower security level. Thus, there is a tradeoffbetween the security level and the energy consumption ofthe H-sensor. However, at high speeds, i.e., 16 m/s, Tbackof fshould be less than 1 second since the number of cluster keyupdates is minimal when Tbackof f is greater than 1 second.However, at low speeds, 1, 2 or 3 seconds are a reasonablechoice for the H-sensors.C. The Effect of TholdFig. 8(b) and Fig. 8(c) show the energy consumption of oneH-sensor and one L-sensor for a pairwise key establishmentin the Random Walk Mobility Model over the course of aday, respectively. The effect of Thold increases as the nodevelocity increases. As Thold increases, the energy consumptiondecreases because in the event that the L-sensors return to theold clusters before the timers expire, no new pairwise masterkey establishment is necessary. As shown in Table III theenergy differences caused by node velocity and Thold is due tothe differences in the frequency of pairwise key establishment.Such frequency is linearly proportional to velocity increases.When Thold ranges from 0 to 500 seconds, energy consumptionrapidly decreases because several moving nodes may return totheir previous clusters within the 500 seconds. However, whenThold ranges from 500 to 1,500 seconds, the energy consumptiondecreases more slowly since the probability of nodesreturning to their previous clusters is dramatically reduced.In the Manhattan Mobility Model, when Thold is small, moreenergy is consumed than in the Random Walk Mobility Modelduring pairwise key establishment since the L-sensors returnto their previous clusters with low frequency. (see Fig. 9(b) andFig. 9(c)). However, when Thold is large, the energy consumedfor the pairwise key establishment dramatically decreases.For instance, as shown in Table IV, when the node speedis 16 m/s and Thold is 1,000 seconds, the number of pairwisekey establishments is only 24,418 which is 5.4 times smaller382 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 10, NO. 2, FEBRUARY 2015TABLE IIITHE FREQUENCY OF PAIRWISE KEY ESTABLISHMENTS FORONE DAY IN RANDOM WALK MOBILITY MODELTABLE IVTHE FREQUENCY OF PAIRWISE KEY ESTABLISHMENTS FORONE DAY IN MANHATTAN MOBILITY MODELthan in the Random Walk Mobility Model with the samesettings.Similarly to Tbackof f , a larger Thold means a lower securitylevel. Thus, Thold should be selected according to VL, energyconsumption amount, and the desired security level. Resultsfrom Fig. 8(c) and Fig. 9(c) show that our scheme is practicalfor real-world monitoring applications such as in animaltracking or traffic monitoring. For example, L-sensors movingat 1 m/s in the Random Walk Mobility Model only use at most0.67 J per day if Thold is greater than 100 seconds. Also,L-sensors moving at 16 m/s in the Manhattan Mobility Modelonly use at most 1.16 J per day if Thold is greater than1,000 seconds. Considering that the average energy of oneC-size alkaline battery is 34,398 J [23], the energyconsumption of the L-sensor for a pairwise key establishmentis relatively small.IX. CONCLUSIONS AND FUTURE WORKSIn this paper, we propose the first certificateless effective keymanagement protocol (CL-EKM) for secure communication indynamic WSNs. CL-EKM supports efficient communicationfor key updates and management when a node leaves or joinsa cluster and hence ensures forward and backward key secrecy.Our scheme is resilient against node compromise, cloningand impersonation attacks and protects the data confidentialityand integrity. The experimental results demonstrate the efficiencyof CL-EKM in resource constrained WSNs. As futurework, we plan to formulate a mathematical model for energyconsumption, based on CL-EKM with various parametersrelated to node movements. This mathematical model will beutilized to estimate the proper value for the Thold and Tbackof fparameters based on the velocity and the desired tradeoffbetween the energy consumption and the security level.

The massive proliferation of wireless infrastructures with complementary characteristics prompts the bandwidth aggregation for Concurrent Multipath Transfer (CMT) over heterogeneous access networks. Stream Control Transmission Protocol (SCTP) is the standard transport-layer solution to enable CMT in multihomed communication environments. However, delivering high-quality streaming video with the existing CMT solutions still remains problematic due to the stringent quality of service (QoS) requirements and path asymmetry in heterogeneous wireless networks.

In this paper, we advance the state of the art by introducing video distortion into the decision process of multipath data transfer. The proposed distortion-aware concurrent multipath transfer (CMT-DA) solution includes three phases: 1) per-path status estimation and congestion control; 2) quality-optimal video flow rate allocation; 3) delay and loss controlled data retransmission. The term ‘flow rate allocation’ indicates dynamically picking appropriate access networks and assigning the transmission rates.

We analytically formulate the data distribution over multiple communication paths to minimize the end-to-end video distortion and derive the solution based on the utility maximization theory. The performance of the proposed CMT-DA is evaluated through extensive semi-physical emulations in Exata involving H.264 video streaming. Experimental results show that CMT-DA outperforms the reference schemes in terms of video peak signal-to-noise ratio (PSNR), good put, and inter-packet delay.

1.2 INTRODUCTION:

During the past few years, mobile video streaming service online gaming, etc. has become one of the “killer applications” and the video traffic headed for hand-held devices has experienced explosive growth. The latest market research conducted by Cisco Company indicates that video streaming accounts for 53 percent of the mobile Internet traffic in parallel, global mobile data is expected to increase 11-fold in the next five years. Another ongoing trend feeding this tremendous growth is the popularity of powerful mobile terminals (e.g., smart phones and iPad), which facilitates individual users to access the Internet and watch videos from everywhere [4].

Despite the rapid advancements in network infrastructures, it is still challenging to deliver high-quality streaming video over wireless platforms. On one hand, the Wi-Fi networks are limited in radio coverage and mobility support for individual users; On the other hand, the cellular networks can well sustain the user mobility but their bandwidth is often inadequate to support the throughput-demanding video applications. Although the 4 G LTE and WiMAX can provide higher peak data rate and extended coverage, the available capacity will still be insufficient compared to the ever-growing video data traffic.

The complementary characteristics of heterogeneous access networks prompt the bandwidth aggregation for concurrent multipath transfer (CMT) to enhance transmission throughput and reliability (see Fig. 1). With the emergency of multihomed/multinetwork terminals CMT is considered to be a promising solution for supporting video streaming in future wireless networking. The key research issue in multihomed video delivery over heterogeneous wireless networks must be effective integration of the limited channel resources available for providing adequate quality of service (QoS). Stream control transmission protocol (SCTP) is the standard transport-layer solution that exploits the multihoming feature to concurrently distribute data across multiple independent end-to-end paths.

Therefore, many CMT solutions have been proposed to optimize the delay, throughput, or reliability performance for efficient data delivery. However, due to the special characteristics of streaming video, these network-level criteria cannot always improve the perceived media quality. For instance, a real-time video application encoded in constant bit rate (CBR) may not effectively leverage the throughput gains since its streaming rate is typically fixed or bounded by the encoding schemes. In addition, involving a communication path with available bandwidth but long delay in the multipath video delivery may degrade the streaming video quality as the end-to-end distortion increases. Consequently, leveraging the CMT for high-quality streaming video over heterogeneous wireless networks is largely unexplored.

In this paper, we investigate the problem by introducing video distortion into the decision process of multipath data transfer over heterogeneous wireless networks. The proposed Distortion-Aware Concurrent Multipath Transfer (CMT-DA) solution is a transport-layer protocol and includes three phases: 1) per-path status estimation and congestion control to exploit the available channel resources; 2) data flow rate allocation to minimize the end-to-end video distortion; 3) delay and loss constrained data retransmission for bandwidth conservation. The detailed descriptions of the proposed solution will be presented in Section 4. Specifically, the contributions of this paper can be summarized in the following.

_ An effective CMT solution that uses path status estimation, flow rate allocation, and retransmission control to optimize the real-time video quality in integrated heterogeneous wireless networks.

_ A mathematical formulation of video data distribution over parallel communication paths to minimize the end-to-end distortion. The utility maximization theory is employed to derive the solution for optimal transmission rate assignment extensive semi-physical emulations in Exata involving real-time H.264 video streaming.

1.3 LITRATURE SURVEY:

CMT-QA: QUALITY-AWARE ADAPTIVE CONCURRENT MULTIPATH DATA TRANSFER IN HETEROGENEOUS WIRELESS NETWORKS

AUTHOR: C. Xu, T. Liu, J. Guan, H. Zhang, and G. M. Muntean,

PUBLICATION: IEEE Trans. Mobile Comput., vol. 12, no. 11, pp. 2193–2205, Nov. 2013.

EXPLANATION:

Mobile devices equipped with multiple network interfaces can increase their throughput by making use of parallel transmissions over multiple paths and bandwidth aggregation, enabled by the stream control transport protocol (SCTP). However, the different bandwidth and delay of the multiple paths will determine data to be received out of order and in the absence of related mechanisms to correct this, serious application-level performance degradations will occur. This paper proposes a novel quality-aware adaptive concurrent multipath transfer solution (CMT-QA) that utilizes SCTP for FTP-like data transmission and real-time video delivery in wireless heterogeneous networks. CMT-QA monitors and analyses regularly each path’s data handling capability and makes data delivery adaptation decisions to select the qualified paths for concurrent data transfer. CMT-QA includes a series of mechanisms to distribute data chunks over multiple paths intelligently and control the data traffic rate of each path independently. CMT-QA’s goal is to mitigate the out-of-order data reception by reducing the reordering delay and unnecessary fast retransmissions. CMT-QA can effectively differentiate between different types of packet loss to avoid unreasonable congestion window adjustments for retransmissions. Simulations show how CMT-QA outperforms existing solutions in terms of performance and quality of service.

PERFORMANCE ANALYSIS OF PROBABILISTIC MULTIPATH TRANSMISSION OF VIDEO STREAMING TRAFFIC OVER MULTI-RADIO WIRELESS DEVICES

AUTHOR: W. Song and W. Zhuang

PUBLICATION: IEEE Trans. Wireless Commun., vol. 11, no. 4, pp. 1554–1564, 2012.

EXPLANATION:

Popular smart wireless devices become equipped with multiple radio interfaces. Multihoming support can be enabled to allow for multiple simultaneous associations with heterogeneous networks. In this study, we focus on video streaming traffic and propose analytical approaches to evaluate the packet-level and call-level performance of a multipath transmission scheme, which sends video traffic bursts over multiple available channels in a probabilistic manner. A probability generation function (PGF) and z-transform method is applied to derive the PGF of packet delay and any arbitrary moment in general. Particularly, we can obtain the average delay, delay jitter, and delay outage probability. The essential characteristics of video traffic are taken into account, such as deterministic burst intervals, highly dynamic burst length, and batch arrivals of transmission packets. The video substream traffic resulting from the probabilistic flow splitting is characterized by means of zero-inflated models. Further, the call-level performance, in terms of flow blocking probability and system throughput, is evaluated with a three-dimensional Markov process and compared with that of an always-best access selection. The numerical and simulations results demonstrate the effectiveness of our analysis framework and the performance gain of multipath transmission.

AN END-TO-END VIRTUAL PATH CONSTRUCTION SYSTEM FOR STABLE LIVE VIDEO STREAMING OVER HETEROGENEOUS WIRELESS NETWORKS

AUTHOR: S. Han, H. Joo, D. Lee, and H. Song

PUBLICATION: IEEE J. Sel. Areas Commun., vol. 29, no. 5, pp. 1032–1041, May 2011.

EXPLANATION:

In this paper, we propose an effective end-to-end virtual path construction system, which exploits path diversity over heterogeneous wireless networks. The goal of the proposed system is to provide a high quality live video streaming service over heterogeneous wireless networks. First, we propose a packetization-aware fountain code to integrate multiple physical paths efficiently and increase the fountain decoding probability over wireless packet switching networks. Second, we present a simple but effective physical path selection algorithm to maximize the effective video encoding rate while satisfying delay and fountain decoding failure rate constraints. The proposed system is fully implemented in software and examined over real WLAN and HSDPA networks.

CHAPTER 2

2.0 SYSTEM ANALYSIS

2.1 EXISTING SYSTEM:

Existing method an effective approach in designing error-resilient wireless video broadcasting systems in recent years, Joint source-channel coding (JSCC) attracts increasing interests in both research community and industry because it shows better results in robust layered video transmission over error-prone channels of various techniques available during these years may be found. However, there are still many open problems in terms of how to serve heterogeneous users with diverse screen features and variable reception performances in wireless video broadcast system. One particular challenging problem of this heterogeneous quality-of-service (QoS) video provision is: the users would prefer flexible video with low quality to match their screens, at the same time; the video stream could be reliable received.

The main technical difficulties are as follows:

• A distinctive characteristic in current wireless broadcast system is that the receivers are highly heterogeneous in terms of their terminal processing capabilities and available bandwidths. In source side, scalable video coding (SVC) has been proposed to provide an attractive solution to this problem.

• However, in order to support flexible video broadcasting, the scalable video sources need to provide adaptation ability through a variety of schemes, such as scalable video stream extraction layer generation with different priority and summarization before they can be transmitted over the error-prone networks.

2.1.1 DISADVANTAGES:

• Existing layered video data is very sensitive to transmission failures, the transmission must be more reliable, have low overhead and support large numbers of devices with heterogeneous characteristics. In broadcast and multicast networks, conventional schemes such as adaptive retransmission have their limitations, for example, retransmission may lead to implosion problem.

• Forward error correction (FEC) and unequal error protection (UEP) are employed to provide the QoS support for video transmission. However, in order to obtain as minimum investment as possible in broadcasting system deployment, server-side must be designed more scalable, reliable, independent, and support vast number of autonomous receivers. Suitable FEC approaches are expected such that can eliminate the retransmission and lower the unnecessary receptions overhead at each receiver-side.

• Conventionally, the joint source and channel coding are designed with seldom consideration in heterogeneous characteristics, and most of the above challenges are ignored in practical video broadcasting system. This leads to the need for heterogeneous QoS video provision in broadcasting network. This paper presents the point of view to study the hybrid-scalable video from new quality metric so as to support users’ heterogeneous requirements.

2.2 PROPOSED SYSTEM:

We proposed Distortion-Aware Concurrent Multipath Transfer (CMT-DA) solution is a transport-layer protocol and includes three phases: 1) per-path status estimation and congestion control to exploit the available channel resources; 2) data flow rate allocation to minimize the end-to-end video distortion; 3) delay and loss constrained data retransmission for bandwidth conservation an effective CMT solution that uses path status estimation, flow rate allocation, and retransmission control to optimize the real-time video quality in integrated heterogeneous wireless networks.

We propose a quality-aware adaptive concurrent multipath transfer (CMT-QA) scheme that distributes the data based on estimated path quality. Although the path status is an important factor that affects the scheduling policy, the application requirements should also be considered to guarantee the QoS. Basically, the proposed CMT-DA is different from the CMT-QA as we take the video distortion as the benchmark. Still, the proposed solutions (path status estimation, flow rate allocation, and retransmission control) in CMT-DA are significantly different from those in CMTQA. In another research conducted by a realistic evaluation tool-set is proposed to analyze and optimize the performance of multimedia distribution when taking advantage of CMT-based multihoming SCTP solutions.

2.2.1 ADVANTAGES:

• We propose a novel out-of-order scheduling approach for in-order arriving of the data chunks in CMT-DA based on the progressive water-filling algorithm. Heterogeneous wireless networks based on fountain code. The encoded multipath streaming model proposed by Chow et al. is a joint multipath and FEC approach for real time live streaming applications.

• We propose an end-to-end virtual path construction system that exploits the path diversity in heterogeneous wireless networks based on fountain code. The encoded multipath streaming model proposed by Chow et al. is a joint multipath and FEC approach for real time live streaming applications. The authors provide asymptotic analysis and derive closed-form solution for the FEC packets allocation.

• The major components at the sender side are the parameter control unit, flow rate allocator, and retransmission controller. The parameter control unit is responsible for processing the acknowledgements (ACKs) feedback from the receiver, estimating the path status and adapting the congestion window size. The delay and loss requirements are imposed by the video applications to achieve the target video quality.
  

2.3 HARDWARE & SOFTWARE REQUIREMENTS:

2.3.1 HARDWARE REQUIREMENT:

v    Processor                                 –    Pentium –IV

• Speed                                      –    1.1 GHz
  • RAM                                       –    256 MB (min)
  • Hard Disk                               –   20 GB
  • Floppy Drive                           –    1.44 MB
  • Key Board                              –    Standard Windows Keyboard
  • Mouse                                     –    Two or Three Button Mouse
  • Monitor                                   –    SVGA

 

2.3.2 SOFTWARE REQUIREMENTS:

• Operating System                   :           Windows XP or Win7
• Front End                                :           JAVA JDK 1.7
• Tools                                       :           Netbeans or Eclipse
• Script                                       :           Java Script
• Document                               :           MS-Office 2007

CHAPTER 3

3.0 SYSTEM DESIGN:

Data Flow Diagram / Use Case Diagram / Flow Diagram:

• The DFD is also called as bubble chart. It is a simple graphical formalism that can be used to represent a system in terms of the input data to the system, various processing carried out on these data, and the output data is generated by the system

• The data flow diagram (DFD) is one of the most important modeling tools. It is used to model the system components. These components are the system process, the data used by the process, an external entity that interacts with the system and the information flows in the system.

• DFD shows how the information moves through the system and how it is modified by a series of transformations. It is a graphical technique that depicts information flow and the transformations that are applied as data moves from input to output.

• DFD is also known as bubble chart. A DFD may be used to represent a system at any level of abstraction. DFD may be partitioned into levels that represent increasing information flow and functional detail.

NOTATION:

SOURCE OR DESTINATION OF DATA:

External sources or destinations, which may be people or organizations or other entities

DATA SOURCE:

Here the data referenced by a process is stored and retrieved.

PROCESS:

People, procedures or devices that produce data’s in the physical component is not identified.

DATA FLOW:

Data moves in a specific direction from an origin to a destination. The data flow is a “packet” of data.

MODELING RULES:

There are several common modeling rules when creating DFDs:

1. All processes must have at least one data flow in and one data flow out.
2. All processes should modify the incoming data, producing new forms of outgoing data.
3. Each data store must be involved with at least one data flow.
4. Each external entity must be involved with at least one data flow.
5. A data flow must be attached to at least one process.

3.1 ARCHITECTURE DIAGRAM

3.2 DATAFLOW DIAGRAM

UML DIAGRAMS:

3.2 USE CASE DIAGRAM:

3.3 CLASS DIAGRAM:

3.4 SEQUENCE DIAGRAM:

3.5 ACTIVITY DIAGRAM:

CHAPTER 4

4.0 IMPLEMENTATION:

4.1 ALGORITHM

4.2 MODULES:

4.3 MODULE DESCRIPTION:

CHAPTER 5

5.0 SYSTEM STUDY:

5.1 FEASIBILITY STUDY:

The feasibility of the project is analyzed in this phase and business proposal is put forth with a very general plan for the project and some cost estimates. During system analysis the feasibility study of the proposed system is to be carried out. This is to ensure that the proposed system is not a burden to the company.  For feasibility analysis, some understanding of the major requirements for the system is essential.

Three key considerations involved in the feasibility analysis are 

• ECONOMICAL FEASIBILITY
• TECHNICAL FEASIBILITY
• SOCIAL FEASIBILITY

5.1.1 ECONOMICAL FEASIBILITY:     

This study is carried out to check the economic impact that the system will have on the organization. The amount of fund that the company can pour into the research and development of the system is limited. The expenditures must be justified. Thus the developed system as well within the budget and this was achieved because most of the technologies used are freely available. Only the customized products had to be purchased.

 

5.1.2 TECHNICAL FEASIBILITY   

This study is carried out to check the technical feasibility, that is, the technical requirements of the system. Any system developed must not have a high demand on the available technical resources. This will lead to high demands on the available technical resources. This will lead to high demands being placed on the client. The developed system must have a modest requirement, as only minimal or null changes are required for implementing this system.  

5.1.3 SOCIAL FEASIBILITY:  

The aspect of study is to check the level of acceptance of the system by the user. This includes the process of training the user to use the system efficiently. The user must not feel threatened by the system, instead must accept it as a necessity. The level of acceptance by the users solely depends on the methods that are employed to educate the user about the system and to make him familiar with it. His level of confidence must be raised so that he is also able to make some constructive criticism, which is welcomed, as he is the final user of the system.

5.2 SYSTEM TESTING:

Testing is a process of checking whether the developed system is working according to the original objectives and requirements. It is a set of activities that can be planned in advance and conducted systematically. Testing is vital to the success of the system. System testing makes a logical assumption that if all the parts of the system are correct, the global will be successfully achieved. In adequate testing if not testing leads to errors that may not appear even many months.

This creates two problems, the time lag between the cause and the appearance of the problem and the effect of the system errors on the files and records within the system. A small system error can conceivably explode into a much larger Problem. Effective testing early in the purpose translates directly into long term cost savings from a reduced number of errors. Another reason for system testing is its utility, as a user-oriented vehicle before implementation. The best programs are worthless if it produces the correct outputs.

5.2.1 UNIT TESTING:

Description	Expected result
Test for application window properties.	All the properties of the windows are to be properly aligned and displayed.
Test for mouse operations.	All the mouse operations like click, drag, etc. must perform the necessary operations without any exceptions.

A program represents the logical elements of a system. For a program to run satisfactorily, it must compile and test data correctly and tie in properly with other programs. Achieving an error free program is the responsibility of the programmer. Program  testing  checks  for  two  types  of  errors:  syntax  and  logical. Syntax error is a program statement that violates one or more rules of the language in which it is written. An improperly defined field dimension or omitted keywords are common syntax errors. These errors are shown through error message generated by the computer. For Logic errors the programmer must examine the output carefully.

5.1.2 FUNCTIONAL TESTING:

Functional testing of an application is used to prove the application delivers correct results, using enough inputs to give an adequate level of confidence that will work correctly for all sets of inputs. The functional testing will need to prove that the application works for each client type and that personalization function work correctly.When a program is tested, the actual output is compared with the expected output. When there is a discrepancy the sequence of instructions must be traced to determine the problem.  The process is facilitated by breaking the program into self-contained portions, each of which can be checked at certain key points. The idea is to compare program values against desk-calculated values to isolate the problems.

Description	Expected result
Test for all modules.	All peers should communicate in the group.
Test for various peer in a distributed network framework as it display all users available in the group.	The result after execution should give the accurate result.

5.1. 3 NON-FUNCTIONAL TESTING:

 The Non Functional software testing encompasses a rich spectrum of testing strategies, describing the expected results for every test case. It uses symbolic analysis techniques. This testing used to check that an application will work in the operational environment. Non-functional testing includes:

• Load testing
• Performance testing
• Usability testing
• Reliability testing
• Security testing

5.1.4 LOAD TESTING:

An important tool for implementing system tests is a Load generator. A Load generator is essential for testing quality requirements such as performance and stress. A load can be a real load, that is, the system can be put under test to real usage by having actual telephone users connected to it. They will generate test input data for system test.

Description	Expected result
It is necessary to ascertain that the application behaves correctly under loads when ‘Server busy’ response is received.	Should designate another active node as a Server.

5.1.5 PERFORMANCE TESTING:

Performance tests are utilized in order to determine the widely defined performance of the software system such as execution time associated with various parts of the code, response time and device utilization. The intent of this testing is to identify weak points of the software system and quantify its shortcomings.

Description	Expected result
This is required to assure that an application perforce adequately, having the capability to handle many peers, delivering its results in expected time and using an acceptable level of resource and it is an aspect of operational management.	Should handle large input values, and produce accurate result in a  expected time.

5.1.6 RELIABILITY TESTING:

The software reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time and it is being ensured in this testing. Reliability can be expressed as the ability of the software to reveal defects under testing conditions, according to the specified requirements. It the portability that a software system will operate without failure under given conditions for a given time interval and it focuses on the behavior of the software element. It forms a part of the software quality control team.

Description	Expected result
This is to check that the server is rugged and reliable and can handle the failure of any of the components involved in provide the application.	In case of failure of  the server an alternate server should take over the job.

5.1.7 SECURITY TESTING:

Security testing evaluates system characteristics that relate to the availability, integrity and confidentiality of the system data and services. Users/Clients should be encouraged to make sure their security needs are very clearly known at requirements time, so that the security issues can be addressed by the designers and testers.

Description	Expected result
Checking that the user identification is authenticated.	In case failure it should not be connected in the framework.
Check whether group keys in a tree are shared by all peers.	The peers should know group key in the same group.

5.1.8 WHITE BOX TESTING:

White  box  testing,  sometimes called  glass-box  testing is  a test  case  design method  that  uses  the  control  structure  of the procedural  design  to  derive  test  cases. Using  white  box  testing  method,  the software  engineer  can  derive  test  cases. The White box testing focuses on the inner structure of the software structure to be tested.

Description	Expected result
Exercise all logical decisions on their true and false sides.	All the logical decisions must be valid.
Execute all loops at their boundaries and within their operational bounds.	All the loops must be finite.
Exercise internal data structures to ensure their validity.	All the data structures must be valid.

5.1.9 BLACK BOX TESTING:

Black box testing, also called behavioral testing, focuses on the functional requirements of the software.  That  is,  black  testing  enables  the software engineer  to  derive  sets  of  input  conditions  that  will  fully  exercise  all  functional requirements  for  a  program.  Black box testing is not alternative to white box techniques.  Rather  it  is  a  complementary  approach  that  is  likely  to  uncover  a different  class  of  errors  than  white box  methods. Black box testing attempts to find errors which focuses on inputs, outputs, and principle function of a software module. The starting point of the black box testing is either a specification or code. The contents of the box are hidden and the stimulated software should produce the desired results.

Description	Expected result
To check for incorrect or missing functions.	All the functions must be valid.
To check for interface errors.	The entire interface must function normally.
To check for errors in a data structures or external data base access.	The database updation and retrieval must be done.
To check for initialization and termination errors.	All the functions and data structures must be initialized properly and terminated normally.

All the above system testing strategies are carried out in as the development, documentation and institutionalization of the proposed goals and related policies is essential.

CHAPTER 6

6.0 SOFTWARE DESCRIPTION:

 

6.1 JAVA TECHNOLOGY:

Java technology is both a programming language and a platform.

 

The Java Programming Language

 

The Java programming language is a high-level language that can be characterized by all of the following buzzwords:

• Simple
  • Architecture neutral
  • Object oriented
  • Portable
  • Distributed     
  • High performance
  • Interpreted     
  • Multithreaded
  • Robust
  • Dynamic
  • Secure     

With most programming languages, you either compile or interpret a program so that you can run it on your computer. The Java programming language is unusual in that a program is both compiled and interpreted. With the compiler, first you translate a program into an intermediate language called Java byte codes —the platform-independent codes interpreted by the interpreter on the Java platform. The interpreter parses and runs each Java byte code instruction on the computer. Compilation happens just once; interpretation occurs each time the program is executed. The following figure illustrates how this works.

You can think of Java byte codes as the machine code instructions for the Java Virtual Machine (Java VM). Every Java interpreter, whether it’s a development tool or a Web browser that can run applets, is an implementation of the Java VM. Java byte codes help make “write once, run anywhere” possible. You can compile your program into byte codes on any platform that has a Java compiler. The byte codes can then be run on any implementation of the Java VM. That means that as long as a computer has a Java VM, the same program written in the Java programming language can run on Windows 2000, a Solaris workstation, or on an iMac.

6.2 THE JAVA PLATFORM:

A platform is the hardware or software environment in which a program runs. We’ve already mentioned some of the most popular platforms like Windows 2000, Linux, Solaris, and MacOS. Most platforms can be described as a combination of the operating system and hardware. The Java platform differs from most other platforms in that it’s a software-only platform that runs on top of other hardware-based platforms.

The Java platform has two components:

• The Java Virtual Machine (Java VM)
• The Java Application Programming Interface (Java API)

You’ve already been introduced to the Java VM. It’s the base for the Java platform and is ported onto various hardware-based platforms.

The Java API is a large collection of ready-made software components that provide many useful capabilities, such as graphical user interface (GUI) widgets. The Java API is grouped into libraries of related classes and interfaces; these libraries are known as packages. The next section, What Can Java Technology Do? Highlights what functionality some of the packages in the Java API provide.

The following figure depicts a program that’s running on the Java platform. As the figure shows, the Java API and the virtual machine insulate the program from the hardware.

Native code is code that after you compile it, the compiled code runs on a specific hardware platform. As a platform-independent environment, the Java platform can be a bit slower than native code. However, smart compilers, well-tuned interpreters, and just-in-time byte code compilers can bring performance close to that of native code without threatening portability.

6.3 WHAT CAN JAVA TECHNOLOGY DO?

The most common types of programs written in the Java programming language are applets and applications. If you’ve surfed the Web, you’re probably already familiar with applets. An applet is a program that adheres to certain conventions that allow it to run within a Java-enabled browser.

However, the Java programming language is not just for writing cute, entertaining applets for the Web. The general-purpose, high-level Java programming language is also a powerful software platform. Using the generous API, you can write many types of programs.

An application is a standalone program that runs directly on the Java platform. A special kind of application known as a server serves and supports clients on a network. Examples of servers are Web servers, proxy servers, mail servers, and print servers. Another specialized program is a servlet.

A servlet can almost be thought of as an applet that runs on the server side. Java Servlets are a popular choice for building interactive web applications, replacing the use of CGI scripts. Servlets are similar to applets in that they are runtime extensions of applications. Instead of working in browsers, though, servlets run within Java Web servers, configuring or tailoring the server.

How does the API support all these kinds of programs? It does so with packages of software components that provides a wide range of functionality. Every full implementation of the Java platform gives you the following features:

• The essentials: Objects, strings, threads, numbers, input and output, data structures, system properties, date and time, and so on.
• Applets: The set of conventions used by applets.
• Networking: URLs, TCP (Transmission Control Protocol), UDP (User Data gram Protocol) sockets, and IP (Internet Protocol) addresses.
• Internationalization: Help for writing programs that can be localized for users worldwide. Programs can automatically adapt to specific locales and be displayed in the appropriate language.
• Security: Both low level and high level, including electronic signatures, public and private key management, access control, and certificates.
• Software components: Known as JavaBeans^TM, can plug into existing component architectures.
• Object serialization: Allows lightweight persistence and communication via Remote Method Invocation (RMI).
• Java Database Connectivity (JDBC^TM): Provides uniform access to a wide range of relational databases.

The Java platform also has APIs for 2D and 3D graphics, accessibility, servers, collaboration, telephony, speech, animation, and more. The following figure depicts what is included in the Java 2 SDK.

 

6.4 HOW WILL JAVA TECHNOLOGY CHANGE MY LIFE?

We can’t promise you fame, fortune, or even a job if you learn the Java programming language. Still, it is likely to make your programs better and requires less effort than other languages. We believe that Java technology will help you do the following:

• Get started quickly: Although the Java programming language is a powerful object-oriented language, it’s easy to learn, especially for programmers already familiar with C or C++.
• Write less code: Comparisons of program metrics (class counts, method counts, and so on) suggest that a program written in the Java programming language can be four times smaller than the same program in C++.
• Write better code: The Java programming language encourages good coding practices, and its garbage collection helps you avoid memory leaks. Its object orientation, its JavaBeans component architecture, and its wide-ranging, easily extendible API let you reuse other people’s tested code and introduce fewer bugs.
• Develop programs more quickly: Your development time may be as much as twice as fast versus writing the same program in C++. Why? You write fewer lines of code and it is a simpler programming language than C++.
• Avoid platform dependencies with 100% Pure Java: You can keep your program portable by avoiding the use of libraries written in other languages. The 100% Pure Java^TM Product Certification Program has a repository of historical process manuals, white papers, brochures, and similar materials online.
• Write once, run anywhere: Because 100% Pure Java programs are compiled into machine-independent byte codes, they run consistently on any Java platform.
• Distribute software more easily: You can upgrade applets easily from a central server. Applets take advantage of the feature of allowing new classes to be loaded “on the fly,” without recompiling the entire program.

 

6.5 ODBC:

 

Microsoft Open Database Connectivity (ODBC) is a standard programming interface for application developers and database systems providers. Before ODBC became a de facto standard for Windows programs to interface with database systems, programmers had to use proprietary languages for each database they wanted to connect to. Now, ODBC has made the choice of the database system almost irrelevant from a coding perspective, which is as it should be. Application developers have much more important things to worry about than the syntax that is needed to port their program from one database to another when business needs suddenly change.

Through the ODBC Administrator in Control Panel, you can specify the particular database that is associated with a data source that an ODBC application program is written to use. Think of an ODBC data source as a door with a name on it. Each door will lead you to a particular database. For example, the data source named Sales Figures might be a SQL Server database, whereas the Accounts Payable data source could refer to an Access database. The physical database referred to by a data source can reside anywhere on the LAN.

The ODBC system files are not installed on your system by Windows 95. Rather, they are installed when you setup a separate database application, such as SQL Server Client or Visual Basic 4.0. When the ODBC icon is installed in Control Panel, it uses a file called ODBCINST.DLL. It is also possible to administer your ODBC data sources through a stand-alone program called ODBCADM.EXE. There is a 16-bit and a 32-bit version of this program and each maintains a separate list of ODBC data sources.

From a programming perspective, the beauty of ODBC is that the application can be written to use the same set of function calls to interface with any data source, regardless of the database vendor. The source code of the application doesn’t change whether it talks to Oracle or SQL Server. We only mention these two as an example. There are ODBC drivers available for several dozen popular database systems. Even Excel spreadsheets and plain text files can be turned into data sources. The operating system uses the Registry information written by ODBC Administrator to determine which low-level ODBC drivers are needed to talk to the data source (such as the interface to Oracle or SQL Server). The loading of the ODBC drivers is transparent to the ODBC application program. In a client/server environment, the ODBC API even handles many of the network issues for the application programmer.

The advantages of this scheme are so numerous that you are probably thinking there must be some catch. The only disadvantage of ODBC is that it isn’t as efficient as talking directly to the native database interface. ODBC has had many detractors make the charge that it is too slow. Microsoft has always claimed that the critical factor in performance is the quality of the driver software that is used. In our humble opinion, this is true. The availability of good ODBC drivers has improved a great deal recently. And anyway, the criticism about performance is somewhat analogous to those who said that compilers would never match the speed of pure assembly language. Maybe not, but the compiler (or ODBC) gives you the opportunity to write cleaner programs, which means you finish sooner. Meanwhile, computers get faster every year.

6.6 JDBC:

In an effort to set an independent database standard API for Java; Sun Microsystems developed Java Database Connectivity, or JDBC. JDBC offers a generic SQL database access mechanism that provides a consistent interface to a variety of RDBMSs. This consistent interface is achieved through the use of “plug-in” database connectivity modules, or drivers. If a database vendor wishes to have JDBC support, he or she must provide the driver for each platform that the database and Java run on.

To gain a wider acceptance of JDBC, Sun based JDBC’s framework on ODBC. As you discovered earlier in this chapter, ODBC has widespread support on a variety of platforms. Basing JDBC on ODBC will allow vendors to bring JDBC drivers to market much faster than developing a completely new connectivity solution.

JDBC was announced in March of 1996. It was released for a 90 day public review that ended June 8, 1996. Because of user input, the final JDBC v1.0 specification was released soon after.

The remainder of this section will cover enough information about JDBC for you to know what it is about and how to use it effectively. This is by no means a complete overview of JDBC. That would fill an entire book.

 

6.7 JDBC Goals:

Few software packages are designed without goals in mind. JDBC is one that, because of its many goals, drove the development of the API. These goals, in conjunction with early reviewer feedback, have finalized the JDBC class library into a solid framework for building database applications in Java.

The goals that were set for JDBC are important. They will give you some insight as to why certain classes and functionalities behave the way they do. The eight design goals for JDBC are as follows:

SQL Level API

The designers felt that their main goal was to define a SQL interface for Java. Although not the lowest database interface level possible, it is at a low enough level for higher-level tools and APIs to be created. Conversely, it is at a high enough level for application programmers to use it confidently. Attaining this goal allows for future tool vendors to “generate” JDBC code and to hide many of JDBC’s complexities from the end user.

SQL Conformance

SQL syntax varies as you move from database vendor to database vendor. In an effort to support a wide variety of vendors, JDBC will allow any query statement to be passed through it to the underlying database driver. This allows the connectivity module to handle non-standard functionality in a manner that is suitable for its users.

JDBC must be implemental on top of common database interfaces

The JDBC SQL API must “sit” on top of other common SQL level APIs. This goal allows JDBC to use existing ODBC level drivers by the use of a software interface. This interface would translate JDBC calls to ODBC and vice versa.

1. Provide a Java interface that is consistent with the rest of the Java system

Because of Java’s acceptance in the user community thus far, the designers feel that they should not stray from the current design of the core Java system.

• Keep it simple

This goal probably appears in all software design goal listings. JDBC is no exception. Sun felt that the design of JDBC should be very simple, allowing for only one method of completing a task per mechanism. Allowing duplicate functionality only serves to confuse the users of the API.

• Use strong, static typing wherever possible

Strong typing allows for more error checking to be done at compile time; also, less error appear at runtime.

• Keep the common cases simple

Because more often than not, the usual SQL calls used by the programmer are simple SELECT’s, INSERT’s, DELETE’s and UPDATE’s, these queries should be simple to perform with JDBC. However, more complex SQL statements should also be possible.

Finally we decided to precede the implementation using Java Networking.

And for dynamically updating the cache table we go for MS Access database.

Java ha two things: a programming language and a platform.

Java is a high-level programming language that is all of the following

Simple                                     Architecture-neutral

Object-oriented                       Portable

Distributed                              High-performance

Interpreted                              Multithreaded

Robust                                     Dynamic Secure

Java is also unusual in that each Java program is both compiled and interpreted. With a compile you translate a Java program into an intermediate language called Java byte codes the platform-independent code instruction is passed and run on the computer.

Compilation happens just once; interpretation occurs each time the program is executed. The figure illustrates how this works.

6.7 NETWORKING TCP/IP STACK:

The TCP/IP stack is shorter than the OSI one:

TCP is a connection-oriented protocol; UDP (User Datagram Protocol) is a connectionless protocol.

IP datagram’s:

The IP layer provides a connectionless and unreliable delivery system. It considers each datagram independently of the others. Any association between datagram must be supplied by the higher layers. The IP layer supplies a checksum that includes its own header. The header includes the source and destination addresses. The IP layer handles routing through an Internet. It is also responsible for breaking up large datagram into smaller ones for transmission and reassembling them at the other end.

UDP:

UDP is also connectionless and unreliable. What it adds to IP is a checksum for the contents of the datagram and port numbers. These are used to give a client/server model – see later.

TCP:

TCP supplies logic to give a reliable connection-oriented protocol above IP. It provides a virtual circuit that two processes can use to communicate.

Internet addresses

In order to use a service, you must be able to find it. The Internet uses an address scheme for machines so that they can be located. The address is a 32 bit integer which gives the IP address.

Network address:

Class A uses 8 bits for the network address with 24 bits left over for other addressing. Class B uses 16 bit network addressing. Class C uses 24 bit network addressing and class D uses all 32.

Subnet address:

Internally, the UNIX network is divided into sub networks. Building 11 is currently on one sub network and uses 10-bit addressing, allowing 1024 different hosts.

Host address:

8 bits are finally used for host addresses within our subnet. This places a limit of 256 machines that can be on the subnet.

Total address:

The 32 bit address is usually written as 4 integers separated by dots.

Port addresses

A service exists on a host, and is identified by its port. This is a 16 bit number. To send a message to a server, you send it to the port for that service of the host that it is running on. This is not location transparency! Certain of these ports are “well known”.

Sockets:

A socket is a data structure maintained by the system to handle network connections. A socket is created using the call socket. It returns an integer that is like a file descriptor. In fact, under Windows, this handle can be used with Read File and Write File functions.

#include <sys/types.h>

#include <sys/socket.h>

int socket(int family, int type, int protocol);

Here “family” will be AF_INET for IP communications, protocol will be zero, and type will depend on whether TCP or UDP is used. Two processes wishing to communicate over a network create a socket each. These are similar to two ends of a pipe – but the actual pipe does not yet exist.

6.8 JFREE CHART:

JFreeChart is a free 100% Java chart library that makes it easy for developers to display professional quality charts in their applications. JFreeChart’s extensive feature set includes:

A consistent and well-documented API, supporting a wide range of chart types;

A flexible design that is easy to extend, and targets both server-side and client-side applications;

Support for many output types, including Swing components, image files (including PNG and JPEG), and vector graphics file formats (including PDF, EPS and SVG);

JFreeChart is “open source” or, more specifically, free software. It is distributed under the terms of the GNU Lesser General Public Licence (LGPL), which permits use in proprietary applications.

 

6.8.1. Map Visualizations:

Charts showing values that relate to geographical areas. Some examples include: (a) population density in each state of the United States, (b) income per capita for each country in Europe, (c) life expectancy in each country of the world. The tasks in this project include: Sourcing freely redistributable vector outlines for the countries of the world, states/provinces in particular countries (USA in particular, but also other areas);

Creating an appropriate dataset interface (plus default implementation), a rendered, and integrating this with the existing XYPlot class in JFreeChart; Testing, documenting, testing some more, documenting some more.

6.8.2. Time Series Chart Interactivity

Implement a new (to JFreeChart) feature for interactive time series charts — to display a separate control that shows a small version of ALL the time series data, with a sliding “view” rectangle that allows you to select the subset of the time series data to display in the main chart.

6.8.3. Dashboards

There is currently a lot of interest in dashboard displays. Create a flexible dashboard mechanism that supports a subset of JFreeChart chart types (dials, pies, thermometers, bars, and lines/time series) that can be delivered easily via both Java Web Start and an applet.

 

6.8.4. Property Editors

The property editor mechanism in JFreeChart only handles a small subset of the properties that can be set for charts. Extend (or reimplement) this mechanism to provide greater end-user control over the appearance of the charts.

CHAPTER 7

APPENDIX

7.1 SAMPLE SOURCE CODE

7.2 SAMPLE OUTPUT

CHAPTER 8

8.1 CONCLUSION:

The future wireless environment is expected to be a converged system that incorporates different access networks with diverse transmission features and capabilities. The increasing powerfulness and popularity of multihomed mobile terminals facilitate the bandwidth aggregation for enhanced transmission reliability and data throughput. Optimizing the SCTP is a critical step towards integrating heterogeneous wireless networks for efficient video delivery.

This paper proposes a novel distortion-aware concurrent multipath transfer scheme to support high-quality video streaming over heterogeneous wireless networks. Through modeling and analysis, we have developed solutions for per-path status estimation, congestion window adaption, flow rate allocation, and data retransmission. As future work, we will study the cost minimization problem of utilizing CMT for mobile video delivery in heterogeneous wireless networks.

Abstract:

The timing channel is a logical communication channel in which information is encoded in the timing between events. Recently, the use of the timing channel has been proposed as a countermeasure to reactive jamming attacks performed by an energy- constrained malicious node. In fact, while a jammer is able to disrupt the information contained in the attacked packets, timing information cannot be jammed, and therefore, timing channels can be exploited to deliver information to the receiver even on a jammed channel. Since the nodes under attack and the jammer have conflicting interests, their interactions can be modeled by means of game theory. Accordingly, in this paper, a game-theoretic model of the interactions between nodes exploiting the timing channel to achieve resilience to jamming attacks and a jammer is derived and analyzed. More specifically, the Nash equilibrium is studied in terms of existence, uniqueness, and convergence under best response dynamics. Furthermore, the case in which the communication nodes set their strategy and the jammer reacts accordingly is modeled and analyzed as a Stackelberg game, by considering both perfect and imperfect knowledge of the jammer’s utility function. Extensive numerical results are presented, showing the impact of network parameters on the system performance.

Introduction:

A timing channel is a communication channel which exploits silence intervals between consecutive transmissions to encode information. Recently, use of timing channels has been proposed in the wireless domain to support low rate, energy efficient communications  as well as covert and resilient communications Timing channels are more although not totally  immune from reactive jamming attacks. In fact, the interfering signal begins its disturbing action against the communication only after identifying an ongoing transmission, and thus after the timing information has been decoded by the receiver.

Timing channel-based communication scheme has been proposed to counteract jamming by establishing a low rate physical layer on top of the traditional physical/link layers using detection and timing of failed packet receptions at the receiver.

The energy cost of jamming the timing channel and the resulting trade-offs have been analyzed. The interactions between the jammer and the node whose transmissions are under attack, which we call target node.

Specifically, assume that the target node wants to maximize the amount of information that can be transmitted per unit of time by means of the timing channel, whereas, the jammer wants to minimize such amount of information while reducing the energy expenditure.

The target node and the jammer have conflicting interests; we develop a game theoretical framework that models their interactions. We investigate both the case in which these two adversaries play their strategies.

 The situation when the target node (the leader) anticipates the actions of the jammer (the follower). To this purpose, we study both the Nash Equilibria (NEs) and Stackelberg Equilibria (SEs) of our proposed games.

Existing system:

Recently, use of timing channels has been proposed in the wireless domain to support low rate, energy efficient communications as well as covert and resilient communications. In existing system methodologies to detect jamming attacks are illustrated; it is also shown that it is possible to identify which kind of jamming attack is ongoing by looking at the signal strength and other relevant network parameters, such as bit and packet errors. Several solutions against reactive jamming have been proposed that exploit different techniques, such as frequency hopping, power control and UN jammed bits.

Disadvantages:

• Continuous jamming is very costly in terms of energy consumption for the jammer
• Existing solutions usually rely on users’ cooperation and coordination, which might not be guaranteed in a jammed environment. In fact, the reactive jammer can totally disrupt each transmitted packet and, consequently, no information can be decoded and then used to this purpose.

Proposed system:

Our proposed system implementation focus on the resilience of timing channels to jamming attacks. In general, these attacks can completely disrupt communications when the jammer continuously emits a high power disturbing signal, i.e., when continuous jammingis performed.

Analyze the interactions between the jammer and the node whose transmissions are under attack, which we call target node. Specifically, we assume that the target node wants to maximize the amount of information that can be transmitted per unit of time by means of the timing channel, whereas, the jammer wants to minimize such amount of information while reducing the energy expenditure.

As the target node and the jammer have conflicting interests, we develop a game theoretical framework that models their interactions. We investigate both the case in which these two adversaries play their strategies simultaneously and the situation when the target node (the leader) anticipates the actions of the jammer (the follower). To this purpose, we study both the Nash Equilibria (NEs) and Stackelberg Equilibria (SEs) of our proposed games.

Advantages:

• System model the interactions between a jammer and a target node as a jamming game
• We prove the existence, uniqueness and convergence to the Nash equilibrium (NE) under best response dynamics
• We prove the existence and uniqueness of the equilibrium of the Stackelberg game where the target node plays as a leader and the jammer reacts consequently
• We investigate in this latter Stackelberg scenario the impact on the achievable performance of imperfect knowledge of the jammer’s utility function;
• We conduct an extensive numerical analysis which shows that our proposed models well capture the main factors behind the utilization of timing channels, thus representing a promising framework for the design and understanding of such systems.

Modules:

NASH Equilibrium Analysis:

The Nash Equilibrium points (NEs), in which both players achieve their highest utility given the strategy profile of the opponent. In the following we also provide proofs of the existence, uniqueness and convergence to the Nash Equilibrium under best response dynamics.

Existence of the Nash Equilibrium:

 It is well known that the intersection points between bT(y) and bJ(x)are the NEs of the game. Therefore, to demonstrate the existence of at least one NE, it suffices to prove that bT(y) and bJ(x) have one or more intersection points. In other words, it is sufficient to find one or more pairs.

Uniqueness of the Nash Equilibrium:

After proving the NE existence in Theorem, let us prove the uniqueness of the NE, that is, there is only one strategy profile such that no player has incentive to deviate unilaterally.

Convergence to the Nash Equilibrium:

Analyze the convergence of the game to the NE when players follow Best Response Dynamics (BRD). In BRD the game starts from any initial point(x(0),y(0))∈Sand, at each successive step, each player plays its strategy by following its best response function.

Performance Analysis

The game allows the leader to achieve a utility which is atleast equal to the utility achieved in the ordinary game at the NE, if we assume perfect knowledge, that is, the target node is completely aware of the utility function of the jammer and its parameters, and thus it is able to evaluate bJ(x). Whereas, if some parameters in the utility function of the jammer are unknown at the target node

Conclusion:

Our system implementation proposed a game-theoretic model of the interactions between a jammer and a communication node that exploits a timing channel to improve resilience to jamming attacks. Structural properties of the utility functions of the two players have been analyzed and exploited to prove the existence and uniqueness of the Nash Equilibrium. The convergence of the game to the Nash Equilibrium has been studied and proved by analyzing the best response dynamics. Furthermore, as the reactive jammer is assumed to start transmitting its interference signal only after detecting activity of the node under attack, a Stackelberg game has been properly investigated, and proofs on the existence and uniqueness of the Stackelberg Equilibrium has been provided.

In this paper, we will focus on the security of advanced metering infrastructure (AMI), which is one of the most crucial components of SG. AMI serves as a bridge for providing bidirectional information flow between user domain and utility domain. AMI’s main functionalities encompass power measurement facilities, assisting adaptive power pricing and demand side management, providing self-healing ability, and interfaces for other systems.

AMI is usually composed of three major types of components, namely, smart meter, data concentrator, and central system (a.k.a. AMI headend) and bidirectional communication networks among those components. AMI is exposed to various security threats such as privacy breach, energy theft, illegal monetary gain, and other malicious activities. As AMI is directly related to revenue earning, customer power consumption, and privacy, of utmost importance is securing its infrastructure. In order to protect AMI from malicious attacks, we look into the intrusion detection system (IDS) aspect of security solution.

We can define IDS as a monitoring system for detecting any unwanted entity into a targeted system (such as AMI in our context). We treat IDS as a second line security measure after the first line of primary AMI security techniques such as encryption, authorization, and authentication, Hence, changing specifications in all key IDS sensors would be expensive and cumbersome. In this paper, we choose to employ anomaly-based IDS using data mining approaches.

1.2 INTRODUCTION

Smart grid (SG) is a set of technologies that integrate modern information technologies with present power grid system. Along with many other benefits, two-way communication, updating users about their consuming behavior, controlling home appliances and other smart components remotely, and monitoring power grid’s stability are unique features of SG. To facilitate such kinds of novel features, SG needs to incorporate many new devices and services. For communicating, monitoring, and controlling of these devices/services, there may also be a need for many new protocols and standards. However, the combination of all these new devices, services, protocols, and standards make SG a very complex system that is vulnerable to increased security threats—like any other complex systems are. In particular, because of its bidirectional, interoperable, and software-oriented nature, SG is very prone to cyber attacks. If proper security measures are not taken, a cyber attack on SG can potentially bring about a huge catastrophic impact on the whole grid and, thus, to the society. Thus, cyber security in SG is treated as one of the vital issues by the National Institute of Standards and Technology and the Federal Energy Regulatory Commission.

In this paper, we will focus on the security of advanced metering infrastructure (AMI), which is one of the most crucial components of SG. AMI serves as a bridge for providing bidirectional information flow between user domain and utility domain [2]. AMI’s main functionalities encompass power measurement facilities, assisting adaptive power pricing and demand side management, providing self-healing ability, and interfaces for other systems. AMI is usually composed of three major types of components, namely, smart meter, data concentrator, and central system (a.k.a. AMI headend) and bidirectional communication networks among those components. Being a complex system in itself, AMI is exposed to various security threats such as privacy breach, energy theft, illegal monetary gain, and other malicious activities. As AMI is directly related to revenue earning, customer power consumption, and privacy, of utmost importance is securing its infrastructure.

1.3 LITRATURE SURVEY

EFFICIENT AUTHENTICATION SCHEME FOR DATA AGGREGATION IN SMART GRID WITH FAULT TOLERANCE AND FAULT DIAGNOSIS

PUBLISH: IEEE Power Energy Soc. Conf. ISGT, 2012, pp. 1–8.

AUTOHR: D. Li, Z. Aung, J. R. Williams, and A. Sanchez

EXPLANATION:

Authentication schemes relying on per-packet signature and per-signature verification introduce heavy cost for computation and communication. Due to its constraint resources, smart grid’s authentication requirement cannot be satisfied by this scheme. Most importantly, it is a must to underscore smart grid’s demand for high availability. In this paper, we present an efficient and robust approach to authenticate data aggregation in smart grid via deploying signature aggregation, batch verification and signature amortization schemes to less communication overhead, reduce numbers of signing and verification operations, and provide fault tolerance. Corresponding fault diagnosis algorithms are contributed to pinpoint forged or error signatures. Both experimental result and performance evaluation demonstrate our computational and communication gains.

CYBER SECURITY ISSUES FOR ADVANCED METERING INFRASTRUCTURE (AMI)

PUBLISH: IEEE Power Energy Soc. Gen. Meet. – Convers. Del. Electr. Energy 21st Century, 2008, pp. 1–5.

AUTOHR: F. M. Cleveland

EXPLANATION:

Advanced Metering Infrastructure (AMI) is becoming of increasing interest to many stakeholders, including utilities, regulators, energy markets, and a society concerned about conserving energy and responding to global warming. AMI technologies, rapidly overtaking the earlier Automated Meter Reading (AMR) technologies, are being developed by many vendors, with portions being developed by metering manufacturers, communications providers, and back-office Meter Data Management (MDM) IT vendors. In this flurry of excitement, very little effort has yet been focused on the cyber security of AMI systems. The comment usually is “Oh yes, we will encrypt everything – that will make everything secure.” That comment indicates unawareness of possible security threats of AMI – a technology that will reach into a large majority of residences and virtually all commercial and industrial customers. What if, for instance, remote connect/disconnect were included as one AMI capability – a function of great interest to many utilities as it avoids truck rolls. What if a smart kid hacker in his basement cracked the security of his AMI system, and sent out 5 million disconnect commands to all customer meters on the AMI system.

INTRUSION DETECTION FOR ADVANCED METERING INFRASTRUCTURES: REQUIREMENTS AND ARCHITECTURAL DIRECTIONS

PUBLISH: IEEE Int. Conf. SmartGridComm, 2010, pp. 350–355.

AUTOHR: R. Berthier, W. H. Sanders, and H. Khurana,

EXPLANATION:

The security of Advanced Metering Infrastructures (AMIs) is of critical importance. The use of secure protocols and the enforcement of strong security properties have the potential to prevent vulnerabilities from being exploited and from having costly consequences. However, as learned from experiences in IT security, prevention is one aspect of a comprehensive approach that must also include the development of a complete monitoring solution. In this paper, we explore the practical needs for monitoring and intrusion detection through a thorough analysis of the different threats targeting an AMI. In order to protect AMI from malicious attacks, we look into the intrusion detection system (IDS) aspect of security solution. We can define IDS as a monitoring system for detecting any unwanted entity into a targeted system (such as AMI in our context). We treat IDS as a second line security measure after the first line of primary AMI security techniques such as encryption, authorization, and authentication, such as [3]. However, Cleveland [4] stressed that these first line security solutions alone are not sufficient for securing AMI.

MOA: MASSIVE ONLINE ANALYSIS, A FRAMEWORK FOR STREAM CLASSIFICATION AND CLUSTERING

PUBLISH: JMLR Workshop Conf. Proc., Workshop Appl. Pattern Anal., 2010, vol. 11, pp. 44–50.

AUTOHR: A. Bifet, G. Holmes, B. Pfahringer, P. Kranen, H. Kremer, T. Jansen, and T. Seidl

EXPLANATION:

In today’s applications, massive, evolving data streams are ubiquitous. Massive Online Analysis (MOA) is a software environment for implementing algorithms and running experiments for online learning from evolving data streams. MOA is designed to deal with the challenging problems of scaling up the implementation of state of the art algorithms to real world dataset sizes and of making algorithms comparable in benchmark streaming settings. It contains a collection of offline and online algorithms for both classification and clustering as well as tools for evaluation. Researchers benefit from MOA by getting insights into workings and problems of different approaches, practitioners can easily compare several algorithms and apply them to real world data sets and settings. MOA supports bi-directional interaction with WEKA, the Waikato Environment for Knowledge Analysis, and is released under the GNU GPL license. Besides providing algorithms and measures for evaluation and comparison, MOA is easily extensible with new contributions and allows the creation of benchmark scenarios through storing and sharing setting files.

SECURING ADVANCED METERING INFRASTRUCTURE USING INTRUSION DETECTION SYSTEM WITH DATA STREAM MINING

PUBLISH: Proc. PAISI, 2012, vol. 7299, pp. 96–111.

AUTOHR: M. A. Faisal, Z. Aung, J. Williams, and A. Sanchez

EXPLANATION:

Advanced metering infrastructure (AMI) is an imperative component of the smart grid, as it is responsible for collecting, measuring, analyzing energy usage data, and transmitting these data to the data concentrator and then to a central system in the utility side. Therefore, the security of AMI is one of the most demanding issues in the smart grid implementation. In this paper, we propose an intrusion detection system (IDS) architecture for AMI which will act as a complimentary with other security measures. This IDS architecture consists of three local IDSs placed in smart meters, data concentrators, and central system (AMI headend). For detecting anomaly, we use data stream mining approach on the public KDD CUP 1999 data set for analysis the requirement of the three components in AMI. From our result and analysis, it shows stream data mining technique shows promising potential for solving security issues in AMI.

DATA STREAM MINING ARCHITECTURE FOR NETWORK INTRUSION DETECTION

PUBLISH: IEEE Int. Conf. IRI, 2004, pp. 363–368

AUTOHR: N. C. N. Chu, A. Williams, R. Alhajj, and K. Barker

EXPLANATION:

In this paper, we propose a stream mining architecture which is based on a single-pass approach. Our approach can be used to develop efficient, effective, and active intrusion detection mechanisms which satisfy the near real-time requirements of processing data streams on a network with minimal overhead. The key idea is that new patterns can now be detected on-the-fly. They are flagged as network attacks or labeled as normal traffic, based on the current network trend, thus reducing the false alarm rates prevalent in active network intrusion systems and increasing the low detection rate which characterizes passive approaches.

RESEARCH ON DATA MINING TECHNOLOGIES APPLYING INTRUSION DETECTION

PUBLISH: Proc. IEEE ICEMMS, 2010, pp. 230–233

AUTOHR: Z. Qun and H. Wen-Jie

EXPLANATION:

Intrusion detection is one of network security area of technology main research directions. Data mining technology was applied to network intrusion detection system (NIDS), may automatically discover the new pattern from the massive network data, to reduce the workload of the manual compilation intrusion behavior patterns and normal behavior patterns. This article reviewed the current intrusion detection technology and the data mining technology briefly. Focus on data mining algorithm in anomaly detection and misuse detection of specific applications. For misuse detection, the main study the classification algorithm; for anomaly detection, the main study the pattern comparison and the cluster algorithm. In pattern comparison to analysis deeply the association rules and sequence rules . Finally, has analysed the difficulties which the current data mining algorithm in intrusion detection applications faced at present, and has indicated the next research direction.

AN EMBEDDED INTRUSION DETECTION SYSTEM MODEL FOR APPLICATION PROGRAM

PUBLISH: IEEE PACIIA, 2008, vol. 2, pp. 910–912.

AUTOHR: S. Wu and Y. Chen

EXPLANATION:

Intrusion detection is an effective security mechanism developed in the recent decade. Because of its wide applicability, intrusion detection becomes the key part of the security mechanism. The modern technologies and models in intrusion detection field are categorized and studied. The characters of current practical IDS are introduced. The theories and realization of IDS based on applications are presented. The basic ideas concerned with how to design and realize the embedded IDS for application are proposed.

ACCURACY UPDATED ENSEMBLE FOR DATA STREAMS WITH CONCEPT DRIFT

PUBLISH: Proc. 6th Int. Conf. HAIS Part II, 2011, pp. 155–163.

AUTOHR: D. Brzeziñski and J. Stefanowski

EXPLANATION:

In this paper we study the problem of constructing accurate block-based ensemble classifiers from time evolving data streams. AWE is the best-known representative of these ensembles. We propose a new algorithm called Accuracy Updated Ensemble (AUE), which extends AWE by using online component classifiers and updating them according to the current distribution. Additional modifications of weighting functions solve problems with undesired classifier excluding seen in AWE. Experiments with several evolving data sets show that, while still requiring constant processing time and memory, AUE is more accurate than AWE.

ACTIVE LEARNING WITH EVOLVING STREAMING DATA

PUBLISH: Proc. ECML-PKDD Part III, 2011, pp. 597–612.

AUTOHR: I. liobaitë, A. Bifet, B. Pfahringer, and G. Holmes

EXPLANATION:

In learning to classify streaming data, obtaining the true labels may require major effort and may incur excessive cost. Active learning focuses on learning an accurate model with as few labels as possible. Streaming data poses additional challenges for active learning, since the data distribution may change over time (concept drift) and classifiers need to adapt. Conventional active learning strategies concentrate on querying the most uncertain instances, which are typically concentrated around the decision boundary. If changes do not occur close to the boundary, they will be missed and classifiers will fail to adapt. In this paper we develop two active learning strategies for streaming data that explicitly handle concept drift. They are based on uncertainty, dynamic allocation of labeling efforts over time and randomization of the search space. We empirically demonstrate that these strategies react well to changes that can occur anywhere in the instance space and unexpectedly.

LEARNING FROM TIME-CHANGING DATA WITH ADAPTIVE WINDOWING

PUBLISH: Proc. SIAM Int. Conf. SDM, 2007, pp. 443–448.

AUTOHR: A. Bifet and R. Gavaldà,

EXPLANATION:

We present a new approach for dealing with distribution change and concept drift when learning from data sequences that may vary with time. We use sliding windows whose size, instead of being fixed a priori, is recomputed online according to the rate of change observed from the data in the window itself. This delivers the user or programmer from having to guess a time-scale for change. Contrary to many related works, we provide rigorous guarantees of performance, as bounds on the rates of false positives and false negatives. Using ideas from data stream algorithmics, we develop a time- and memory-efficient version of this algorithm, called ADWIN2. We show how to combine ADWIN2 with the Naïve Bayes (NB) predictor, in two ways: one, using it to monitor the error rate of the current model and declare when revision is necessary and, two, putting it inside the NB predictor to maintain up-to-date estimations of conditional probabilities in the data. We test our approach using synthetic and real data streams and compare them to both fixed-size and variable-size window strategies with good results.

CHAPTER 2

2.0 SYSTEM ANALYSIS

2.1 EXISTING SYSTEM:

Existing methods in protecting an AMI against malicious activities is to create a monitoring solution that covers the heterogeneity of communication technologies through their requirements (e.g., encryption and real time) and constraints (e.g., topology and bandwidth). It is critical to identify these elements, for two reasons: 1) they can help to define the potential impact of malicious activities targeting the AMI; and 2) they can impose limits on the functionalities and security of a monitoring solution. For instance, the fact that large portions of an AMI network are wireless and use a mesh network topology facilitates network-related attacks such as traffic interception, and the design of the monitoring architecture is more challenging than in a traditional wired network. Moreover, a large number of nodes are deployed in the field or in consumer facilities, which means that attacks requiring physical access are easier to conduct.

These detection techniques are different for two fundamental reasons.

First, signature-based IDS uses a blacklist approach, while anomaly- and specification-based IDS use a white list approach. A blacklist approach requires creation of a knowledge base of malicious activity, while a white list approach requires training of the system and identification of its normal or correct behavior list approaches is that they provide little information about the root causes of attacks.

A second fundamental difference lies in the level of understanding required by each approach. Signature- and anomaly-based IDSes belong to the same group by monitoring activity at a low level, while specification-based IDS requires a high-level and stateful understanding of the activity monitored.

2.1.1 DISADVANTAGES:

• Curious eavesdroppers, who are motivated to learn about the activity of their neighbors by listening in on the traffic of the surrounding meters

• Motivated eavesdroppers, who desire to gather information about potential victims as part of an organized theft. • Unethical customers, who are motivated to steal electricity by tampering with the metering equipment installed inside their homes.

• Overly intrusive meter data management agencies, which are motivated to gain high-resolution energy and behavior profiles about their users, which can damage customer privacy. This type of attacker also includes employees who could attempt to spy illegitimately on customers.

• Active attackers, who are motivated by financial gain or terrorist goals. The objective of a terrorist would be to create large-scale disruption of the grid, either by remotely cutting off many customers or by creating instability in the distribution or transmission networks. Active attackers attracted by financial gain could also use disruptive actions, such as Denial of Service (DoS) attacks, or they could develop self-propagating malware in order to create revenue-making.

• Publicity seekers, who use techniques similar to those of other types of attackers, but in a potentially less harmful way, because they are more interested in fame and usually have limited financial resources. Attackers may use a variety of attack techniques to reach their objectives. Based on a survey of the related literature information about the attack consequences and will be used in the next section to identify the monitoring mechanisms required for an intrusion detection system.

2.2 PROPOSED SYSTEM:

We propose a new AMI IDS architecture based on the AMI architecture presented by OPENMeter, which is a project deployed by several European countries to reduce gap between the state-of-the-art technologies and AMI’s requirements. We use the data stream mining algorithms available in the Massive Online Analysis (MOA) in order to simulate the IDSs of the proposed architecture.

Our proposed IDS architecture follows a sequential process. Communication data from various sources are inserted to Acceptor Module. Preprocessing Unit is responsible for producing data according to predetermined attributes by monitoring the communication data. This generated data would be treated as input for Stream Mining Module. Stream Mining Module runs a data stream mining algorithm over the data generated by Preprocessing Unit. Decision Maker Unit decides whether it should trigger an alarm or not.

This module also keeps records of the information associated with attacks. These records will be used for further analysis and improving the attack database. The proposed IDS architecture for the other two types of AMI components, namely, data concentrator and AMI head end, is more or less similar to that of smart meter IDS. Again, the security boxes for those components can be either inside (in the form of software or add-on hardware card) or outside (in the form of a dedicated box or server). In order to simultaneously our proposed IDS architecture follows a sequential process.

Preprocessing Unit is responsible for producing data according to predetermined attributes by monitoring the communication data. This generated data would be treated as input for Stream Mining Module. Stream Mining Module runs a data stream mining algorithm over the data generated by Preprocessing Unit. Decision Maker Unit decides whether it should trigger an alarm or not. This module also keeps records of the information associated with attacks. These records will be used for further analysis and improving the attack database.

2.2.1 ADVANTAGES:

• We have proposed a reliable and pragmatic IDS architecture for AMI.

• We have conducted a set of experiments on a public IDS data set using state-of-the-art data stream mining techniques and observed their performances.

• We have performed a feasibility study of applying these data stream mining algorithms for different components of the proposed IDS architecture. Note that proposing a new data stream mining algorithm is out of the scope of this paper and is planned as future work.

2.3 HARDWARE & SOFTWARE REQUIREMENTS:

2.3.1 HARDWARE REQUIREMENT:

v    Processor                                 –    Pentium –IV

• Speed                                      –    1.1 GHz
  • RAM                                       –    256 MB (min)
  • Hard Disk                               –   20 GB
  • Floppy Drive                           –    1.44 MB
  • Key Board                              –    Standard Windows Keyboard
  • Mouse                                     –    Two or Three Button Mouse
  • Monitor                                   –    SVGA

 

2.3.2 SOFTWARE REQUIREMENTS:

• Operating System                   :           Windows XP or Win7
• Front End                                :           JAVA JDK 1.7
• Document                               :           MS-Office 2007

CHAPTER 3

3.0 SYSTEM DESIGN:

Data Flow Diagram / Use Case Diagram / Flow Diagram:

• The DFD is also called as bubble chart. It is a simple graphical formalism that can be used to represent a system in terms of the input data to the system, various processing carried out on these data, and the output data is generated by the system

• The data flow diagram (DFD) is one of the most important modeling tools. It is used to model the system components. These components are the system process, the data used by the process, an external entity that interacts with the system and the information flows in the system.

• DFD shows how the information moves through the system and how it is modified by a series of transformations. It is a graphical technique that depicts information flow and the transformations that are applied as data moves from input to output.

• DFD is also known as bubble chart. A DFD may be used to represent a system at any level of abstraction. DFD may be partitioned into levels that represent increasing information flow and functional detail.

NOTATION:

SOURCE OR DESTINATION OF DATA:

External sources or destinations, which may be people or organizations or other entities

DATA SOURCE:

Here the data referenced by a process is stored and retrieved.

PROCESS:

People, procedures or devices that produce data’s in the physical component is not identified.

DATA FLOW:

Data moves in a specific direction from an origin to a destination. The data flow is a “packet” of data.

MODELING RULES:

There are several common modeling rules when creating DFDs:

1. All processes must have at least one data flow in and one data flow out.
2. All processes should modify the incoming data, producing new forms of outgoing data.
3. Each data store must be involved with at least one data flow.
4. Each external entity must be involved with at least one data flow.
5. A data flow must be attached to at least one process.

3.1 ARCHITECTURE DIAGRAM

3.2 DATAFLOW DIAGRAM

LEVEL 0

Base station

IP Address

Generate Authentication Key

Send Packet Data

LEVEL 1

Base station

IP Address

Send Data

File transfer

Socket connection

Connecting

LEVEL 2

Router

IP Address

Socket connection

Routing

Verify file transaction

IDS Attack

Hash implementation Authentication key infrastructure Certificate revocation list

Security analysis

Encrypt

LEVEL 3

Node

IP Address

Received path

IDS Attack

Data Received

UML DIAGRAMS:

3.2 USE CASE DIAGRAM:

Base station

Router

IP Address

Socket connection

Data Transfer

Authentication

Hash implementation Public key infrastructure Certificate revocation list

IDS

Received Node

Received

Node

3.3 CLASS DIAGRAM:

Node

IP Adress

Browse received path

Connecting ()

Base station

IP Address

Browse file

Connecting

Socket connection ()  ()

File transfer ()  ()

Router

IP Address

Select Connection

Routing ()

File received ()

Start receiving

Security analysis ()

Encrypt ()

3.4 SEQUENCE DIAGRAM:

Connection established

Send data Generating Authentication  Form routing Routing Finished   Decode data and view   Connection terminate Source Base station        Destination Establish communication Connection established IDS Attack   Data received Routing Success  

3.5 ACTIVITY DIAGRAM:

Node

Router

Check

File not receive

IP address & select connection

Security analysis IDS

Routing

Yes Start file receive

No

IP Address & browse file

IP Address

Browse received path & connecting

File Received

Base station

Socket connection & connecting

File transfer

CHAPTER 4

4.0 IMPLEMENTATION:

AMI:

AMI is an updated version of automatic or automated meter reading (AMR) [2]. Present traditional AMR helps a utility company in reading meters through one-way communication. However, as AMR cannot meet the current requirements for two-way communication and others, AMI is introduced. AMI is composed of smart meters, data concentrators, and central system (AMI headend) and the communication networks among them. These AMI components are usually located in various networks and different realms such as public and private ones. Fig. 1 gives a pictorial view of AMI integration in a broader context of power generation, distribution, etc. From this figure, we can see that the smart meter, responsible for monitoring and recording power usage of home appliances, etc., is the key equipment for consumers.

Home appliances and other integrated devices/systems such as water and gas meters, in-home display, plug-in electric vehicle/plug-in hybrid electric vehicle, smart thermostat, rooftop photovoltaic system, etc., constitute a home area network (HAN), which is connected to the smart meter. For communicating among these constituents, Zegbee or power line communication can be used. A number of individual smart meters communicate to a data concentrator through neighborhood area network (NAN). WiMAX, cellulartechnologies, etc., are possible means for this network. A number of data concentrators are connected to an AMI headend in the utility side using wide area network (WAN). Various long-distance communication technologies such as fiber optic, digital subscriber line, etc., are used in WAN. The AMI headend located in the utility side consists of meter data management system, geographic information system (GIS), configuration system, etc. These subsystems may build a local area network (LAN) for intercommunication.

Let us look at the first component of AMI, namely, the smart meter. Along with the houses of ordinary people, smart meters are also installed in crucial places such as companies, banks, hospitals, educational institutes, government agencies, parliaments, and presidential residences. Thus, the security of smart meters is a vital issue. To our best knowledge, current smart meters do not possess IDS facility yet. If we are to furnish smart meters with IDS, one possible approach is to develop embedded software for IDS, such as the one proposed in [20], and update the firmware of the smart meter to include these embedded IDS. Although this can be done with relative ease, the main problem is the limitation of computing resources in the current smart meters. They are mostly equipped with low-end processors and limited amounts of main memory (in kilobytes to a few megabytes range). Although this may change in the near future, since a good number of smart meters have already been deployed in many developed countries, it is not very easy to replace them orupgrade those existing ones with more powerful resources.

Since a smart meter is supposed to consume most of its processor and main memory resources for its core businesses (such as recording electricity usage, interaction with other smart home appliances, and two-way communication with its associated data concentrator and, ultimately, the headend), only a small fraction of its already limited resources is available for IDS’ data processing purpose. We try to solve this problem of resource scarceness by proposing to use a separate IDS entity, either installed outside the smart meter (for existing ones) or integrated within the smart meter (for new ones). We name such an entity a “security box.” A possible design of a smart meter with this security box is provided in Fig. 2(a) based on the one presented in [22]. Here, we the show security box as a simple meter IDS. However, it is open for this component to cover other security functions such as firewall, encryption, authorization, authentication, etc. Care should be taken that the security box for the smart meter should not be too expensive and hence should be equipped with resources just enough to perform computations for IDS (and other security-related calculations, if applicable) at the meter level.

4.1 ALGORITHM

DATA STREAM MINING ALGORITHMS

 We use MOA in our experiment. It is an open source data stream mining framework. Although there are various static and evolving stream mining classification algorithms available in this software environment, we are only interested inthe evolving ones. Evolving classification algorithms care about the concept change or the distribution change in the data stream. There are 16 evolving data stream classifiers in MOA. After an initial trail on those 16 classifiers, the 7 ensemble classifiers listed in Table II are selected. (From now on, we will write classifier names in MOA in italic. In many cases, MOA’s classifiernames are self-explanatory.) These seven classifiers are chosen because they offered the highest accuracies (evaluated with EvaluatePrequential method in MOA) on the training set. Different variants of a Hoeffding tree are used as the base learner in MOA.

The algorithm establishes the Hoeffding bound, which quantifies the number of observations required to estimate necessary statistics within a prescribed precision. Mathematically, the Hoeffding bound can be expressed using (1), which says that the true mean of a random variable of range R will not differ from estimated mean after n independent examples or observations by more than with probability. The descriptions of these seven selected ensemble learners are as follows. (For more detailed descriptions, refer to the original papers [23]–[30].) AccuracyUpdatedEnsemble (AUE): It is a block-based ensemble classifier, an improved version of Accuracy Weighted Ensemble (AWE), for evolving data stream. Accuracy Updated Ensemble (AUE) makes this enhancement by using online component classifiers, which updates the base classifiers rather than only adjust their weights, as well as updating them according to present distribution. In addition, this method also leverages the drawback of AWE by redefining the weighting function.

4.2 MODULES:

SERVER CLIENT MODULE:

SMART GRID AMI:

DATA STREAM MINING METHODS:

INTRUSION DETECTION SYSTEM (IDS):

4.3 MODULE DESCRIPTION:

SERVER CLIENT MODULE:

Client-server computing or networking is a distributed application architecture that partitions tasks or workloads between service providers (servers) and service requesters, called clients. Often clients and servers operate over a computer network on separate hardware. A server machine is a high-performance host that is running one or more server programs which share its resources with clients. A client also shares any of its resources; Clients therefore initiate communication sessions with servers which await (listen to) incoming requests. Network-accessible resources may be deployed in a network as surveillance and early-warning tools, as the detection of attackers are not normally accessed for legitimate purposes.

Techniques used by the attackers that attempt to compromise these decoy resources are studied during and after an attack to keep an eye on new exploitation techniques. Such analysis may be used to further tighten security of the actual network being protected by the data’s. Data forwarding can also direct an attacker’s attention away from legitimate servers. A user encourages attackers to spend their time and energy on the decoy server while distracting their attention from the data on the real server. Similar to a server, a user is a network set up with intentional vulnerabilities. Its purpose is also to invite attacks so that the attacker’s methods can be studied and that information can be used to increase network security.

SMART GRID AMI:

We developing a specification for AMI networks is effective gradually, fresh specifications need to be included. Hence, changing specifications in all key IDS sensors would be expensive and cumbersome. In this paper, we choose to employ anomaly-based IDS using data mining approaches. However, instead of considering conventional static mining techniques, we select stream mining, precisely “evolving data stream mining,” as this approach is a more realistic approach in real-world monitoring and intrusion detection for AMI as various novel attacks can be introduced in AMI. Rodrigues and Gama mentioned that SG networks have various distributed sources of high-speed data streams. These stream data can be attributed as open ended and high speed and are produced in non stationary distributions. Thus, the dynamics of data are unpredictable. As the number of smart meters is suppose to eventually grow and their roles in AMI evolve over time, the topology of SG networks may change. Thus, data distribution in AMI networks can be changed. Hence, the model should be able to cope with evolved data.

AMI is composed of smart meters, data concentrators, and central system (AMI headend) and the communication networks among them. These AMI components are usually located in various networks and different realms such as public and private ones a pictorial view of AMI integration in a broader context of power generation, distribution, etc. From this figure, we can see that the smart meter, responsible for monitoring and recording power usage of home appliances, etc., is the key equipment for consumers. Home appliances and other integrated devices/systems such as water and gas meters, in-home display, plug-in electric vehicle/plug-in hybrid electric vehicle, smart thermostat, rooftop photovoltaic system, etc., constitute a home area network (HAN), which is connected to the smart meter.

DATA STREAM MINING METHODS:

We will focus on the security of advanced metering infrastructure (AMI), which is one of the most crucial components of SG. AMI serves as a bridge for providing bidirectional information flow between user domain and utility domain. AMI’s main functionalities encompass power measurement facilities, assisting adaptive power pricing and demand side management, providing self-healing ability, and interfaces for other systems.

AMI is usually composed of three major types of components, namely, smart meter, data concentrator, and central system (a.k.a. AMI headend) and bidirectional communication networks among those components. Being a complex system in itself, AMI is exposed to various security threats such as privacy breach, energy theft, illegal monetary gain, and other malicious activities. As AMI is directly related to revenue earning, customer power consumption, and privacy, of utmost importance is securing its infrastructure.

Our data streaming protect AMI from malicious attacks, we look into the intrusion detection system (IDS) aspect of security solution. We can define IDS as a monitoring system for detecting any unwanted entity into a targeted system (such as AMI in our context). We treat IDS as a second line security measure after the first line of primary AMI security techniques such as encryption, authorization, and authentication solutions alone are not sufficient for securing AMI.

INTRUSION DETECTION SYSTEM (IDS):

We develop embedded software for IDS, such as the one proposed in, and update the firmware of the smart meter to include these embedded IDS. Although this can be done with relative ease, the main problem is the limitation of computing resources in the current smart meters. They are mostly equipped with low-end processors and limited amounts of main memory (in kilobytes to a few megabytes range). Although this may change in the near future, since a good number of smart meters have already been deployed in many developed countries, it is not very easy to replace them or upgrade those existing ones with more powerful resources.

Since a smart meter is supposed to consume most of its processor and main memory resources for its core businesses (such as recording electricity usage, interaction with other smart home appliances, and two-way communication with its associated data concentrator and, ultimately, the head end), only a small fraction of its already limited resources is available for IDS’ data processing purpose. We try to solve this problem of resource scarceness by proposing to use a separate IDS entity, either installed outside the smart meter (for existing ones) or integrated within the smart meter (for new ones). We name such an entity a “security box.” A possible design of a smart meter with this security box is provided in Fig. 2(a) based on the one presented in show security box as a simple meter IDS.

However, it is open for this component to cover other security functions such as firewall, encryption, authorization, authentication, etc. IDS architecture for the other two types of AMI components, namely, data concentrator and AMI headend, is more or less similar to that of smart meter IDS. Again, the security boxes for those components can be either inside (in the form of software or add-on hardware card) or outside (in the form of a dedicated box or server). In order to simultaneously monitor a large number of data lows received from a large number of smart meters and detect security threats, such security box hardware (or the host equipment, in the case of software) must be rich in computing resources.

CHAPTER 5

5.0 SYSTEM STUDY:

5.1 FEASIBILITY STUDY:

The feasibility of the project is analyzed in this phase and business proposal is put forth with a very general plan for the project and some cost estimates. During system analysis the feasibility study of the proposed system is to be carried out. This is to ensure that the proposed system is not a burden to the company.  For feasibility analysis, some understanding of the major requirements for the system is essential.

Three key considerations involved in the feasibility analysis are 

• ECONOMICAL FEASIBILITY
• TECHNICAL FEASIBILITY
• SOCIAL FEASIBILITY

5.1.1 ECONOMICAL FEASIBILITY:     

This study is carried out to check the economic impact that the system will have on the organization. The amount of fund that the company can pour into the research and development of the system is limited. The expenditures must be justified. Thus the developed system as well within the budget and this was achieved because most of the technologies used are freely available. Only the customized products had to be purchased.

 

5.1.2 TECHNICAL FEASIBILITY   

This study is carried out to check the technical feasibility, that is, the technical requirements of the system. Any system developed must not have a high demand on the available technical resources. This will lead to high demands on the available technical resources. This will lead to high demands being placed on the client. The developed system must have a modest requirement, as only minimal or null changes are required for implementing this system.  

5.1.3 SOCIAL FEASIBILITY:  

The aspect of study is to check the level of acceptance of the system by the user. This includes the process of training the user to use the system efficiently. The user must not feel threatened by the system, instead must accept it as a necessity. The level of acceptance by the users solely depends on the methods that are employed to educate the user about the system and to make him familiar with it. His level of confidence must be raised so that he is also able to make some constructive criticism, which is welcomed, as he is the final user of the system.

5.2 SYSTEM TESTING:

Testing is a process of checking whether the developed system is working according to the original objectives and requirements. It is a set of activities that can be planned in advance and conducted systematically. Testing is vital to the success of the system. System testing makes a logical assumption that if all the parts of the system are correct, the global will be successfully achieved. In adequate testing if not testing leads to errors that may not appear even many months.

This creates two problems, the time lag between the cause and the appearance of the problem and the effect of the system errors on the files and records within the system. A small system error can conceivably explode into a much larger Problem. Effective testing early in the purpose translates directly into long term cost savings from a reduced number of errors. Another reason for system testing is its utility, as a user-oriented vehicle before implementation. The best programs are worthless if it produces the correct outputs.

5.2.1 UNIT TESTING:

Description	Expected result
Test for application window properties.	All the properties of the windows are to be properly aligned and displayed.
Test for mouse operations.	All the mouse operations like click, drag, etc. must perform the necessary operations without any exceptions.

A program represents the logical elements of a system. For a program to run satisfactorily, it must compile and test data correctly and tie in properly with other programs. Achieving an error free program is the responsibility of the programmer. Program  testing  checks  for  two  types  of  errors:  syntax  and  logical. Syntax error is a program statement that violates one or more rules of the language in which it is written. An improperly defined field dimension or omitted keywords are common syntax errors. These errors are shown through error message generated by the computer. For Logic errors the programmer must examine the output carefully.

5.1.2 FUNCTIONAL TESTING:

Functional testing of an application is used to prove the application delivers correct results, using enough inputs to give an adequate level of confidence that will work correctly for all sets of inputs. The functional testing will need to prove that the application works for each client type and that personalization function work correctly.When a program is tested, the actual output is compared with the expected output. When there is a discrepancy the sequence of instructions must be traced to determine the problem.  The process is facilitated by breaking the program into self-contained portions, each of which can be checked at certain key points. The idea is to compare program values against desk-calculated values to isolate the problems.

Description	Expected result
Test for all modules.	All peers should communicate in the group.
Test for various peer in a distributed network framework as it display all users available in the group.	The result after execution should give the accurate result.

5.1. 3 NON-FUNCTIONAL TESTING:

 The Non Functional software testing encompasses a rich spectrum of testing strategies, describing the expected results for every test case. It uses symbolic analysis techniques. This testing used to check that an application will work in the operational environment. Non-functional testing includes:

• Load testing
• Performance testing
• Usability testing
• Reliability testing
• Security testing

5.1.4 LOAD TESTING:

An important tool for implementing system tests is a Load generator. A Load generator is essential for testing quality requirements such as performance and stress. A load can be a real load, that is, the system can be put under test to real usage by having actual telephone users connected to it. They will generate test input data for system test.

Description	Expected result
It is necessary to ascertain that the application behaves correctly under loads when ‘Server busy’ response is received.	Should designate another active node as a Server.

5.1.5 PERFORMANCE TESTING:

Performance tests are utilized in order to determine the widely defined performance of the software system such as execution time associated with various parts of the code, response time and device utilization. The intent of this testing is to identify weak points of the software system and quantify its shortcomings.

Description	Expected result
This is required to assure that an application perforce adequately, having the capability to handle many peers, delivering its results in expected time and using an acceptable level of resource and it is an aspect of operational management.	Should handle large input values, and produce accurate result in a  expected time.

5.1.6 RELIABILITY TESTING:

The software reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time and it is being ensured in this testing. Reliability can be expressed as the ability of the software to reveal defects under testing conditions, according to the specified requirements. It the portability that a software system will operate without failure under given conditions for a given time interval and it focuses on the behavior of the software element. It forms a part of the software quality control team.

Description	Expected result
This is to check that the server is rugged and reliable and can handle the failure of any of the components involved in provide the application.	In case of failure of  the server an alternate server should take over the job.

5.1.7 SECURITY TESTING:

Security testing evaluates system characteristics that relate to the availability, integrity and confidentiality of the system data and services. Users/Clients should be encouraged to make sure their security needs are very clearly known at requirements time, so that the security issues can be addressed by the designers and testers.

Description	Expected result
Checking that the user identification is authenticated.	In case failure it should not be connected in the framework.
Check whether group keys in a tree are shared by all peers.	The peers should know group key in the same group.

5.1.8 WHITE BOX TESTING:

White  box  testing,  sometimes called  glass-box  testing is  a test  case  design method  that  uses  the  control  structure  of the procedural  design  to  derive  test  cases. Using  white  box  testing  method,  the software  engineer  can  derive  test  cases. The White box testing focuses on the inner structure of the software structure to be tested.

Description	Expected result
Exercise all logical decisions on their true and false sides.	All the logical decisions must be valid.
Execute all loops at their boundaries and within their operational bounds.	All the loops must be finite.
Exercise internal data structures to ensure their validity.	All the data structures must be valid.

5.1.9 BLACK BOX TESTING:

Black box testing, also called behavioral testing, focuses on the functional requirements of the software.  That  is,  black  testing  enables  the software engineer  to  derive  sets  of  input  conditions  that  will  fully  exercise  all  functional requirements  for  a  program.  Black box testing is not alternative to white box techniques.  Rather  it  is  a  complementary  approach  that  is  likely  to  uncover  a different  class  of  errors  than  white box  methods. Black box testing attempts to find errors which focuses on inputs, outputs, and principle function of a software module. The starting point of the black box testing is either a specification or code. The contents of the box are hidden and the stimulated software should produce the desired results.

Description	Expected result
To check for incorrect or missing functions.	All the functions must be valid.
To check for interface errors.	The entire interface must function normally.
To check for errors in a data structures or external data base access.	The database updation and retrieval must be done.
To check for initialization and termination errors.	All the functions and data structures must be initialized properly and terminated normally.

All the above system testing strategies are carried out in as the development, documentation and institutionalization of the proposed goals and related policies is essential.

CHAPTER 6

6.0 SOFTWARE DESCRIPTION:

 

6.1 JAVA TECHNOLOGY:

Java technology is both a programming language and a platform.

 

The Java Programming Language

 

The Java programming language is a high-level language that can be characterized by all of the following buzzwords:

• Simple
  • Architecture neutral
  • Object oriented
  • Portable
  • Distributed     
  • High performance
  • Interpreted     
  • Multithreaded
  • Robust
  • Dynamic
  • Secure     

With most programming languages, you either compile or interpret a program so that you can run it on your computer. The Java programming language is unusual in that a program is both compiled and interpreted. With the compiler, first you translate a program into an intermediate language called Java byte codes —the platform-independent codes interpreted by the interpreter on the Java platform. The interpreter parses and runs each Java byte code instruction on the computer. Compilation happens just once; interpretation occurs each time the program is executed. The following figure illustrates how this works.

You can think of Java byte codes as the machine code instructions for the Java Virtual Machine (Java VM). Every Java interpreter, whether it’s a development tool or a Web browser that can run applets, is an implementation of the Java VM. Java byte codes help make “write once, run anywhere” possible. You can compile your program into byte codes on any platform that has a Java compiler. The byte codes can then be run on any implementation of the Java VM. That means that as long as a computer has a Java VM, the same program written in the Java programming language can run on Windows 2000, a Solaris workstation, or on an iMac.

6.2 THE JAVA PLATFORM:

A platform is the hardware or software environment in which a program runs. We’ve already mentioned some of the most popular platforms like Windows 2000, Linux, Solaris, and MacOS. Most platforms can be described as a combination of the operating system and hardware. The Java platform differs from most other platforms in that it’s a software-only platform that runs on top of other hardware-based platforms.

The Java platform has two components:

• The Java Virtual Machine (Java VM)
• The Java Application Programming Interface (Java API)

You’ve already been introduced to the Java VM. It’s the base for the Java platform and is ported onto various hardware-based platforms.

The Java API is a large collection of ready-made software components that provide many useful capabilities, such as graphical user interface (GUI) widgets. The Java API is grouped into libraries of related classes and interfaces; these libraries are known as packages. The next section, What Can Java Technology Do? Highlights what functionality some of the packages in the Java API provide.

The following figure depicts a program that’s running on the Java platform. As the figure shows, the Java API and the virtual machine insulate the program from the hardware.

Native code is code that after you compile it, the compiled code runs on a specific hardware platform. As a platform-independent environment, the Java platform can be a bit slower than native code. However, smart compilers, well-tuned interpreters, and just-in-time byte code compilers can bring performance close to that of native code without threatening portability.

6.3 WHAT CAN JAVA TECHNOLOGY DO?

The most common types of programs written in the Java programming language are applets and applications. If you’ve surfed the Web, you’re probably already familiar with applets. An applet is a program that adheres to certain conventions that allow it to run within a Java-enabled browser.

However, the Java programming language is not just for writing cute, entertaining applets for the Web. The general-purpose, high-level Java programming language is also a powerful software platform. Using the generous API, you can write many types of programs.

An application is a standalone program that runs directly on the Java platform. A special kind of application known as a server serves and supports clients on a network. Examples of servers are Web servers, proxy servers, mail servers, and print servers. Another specialized program is a servlet.

A servlet can almost be thought of as an applet that runs on the server side. Java Servlets are a popular choice for building interactive web applications, replacing the use of CGI scripts. Servlets are similar to applets in that they are runtime extensions of applications. Instead of working in browsers, though, servlets run within Java Web servers, configuring or tailoring the server.

How does the API support all these kinds of programs? It does so with packages of software components that provides a wide range of functionality. Every full implementation of the Java platform gives you the following features:

• The essentials: Objects, strings, threads, numbers, input and output, data structures, system properties, date and time, and so on.
• Applets: The set of conventions used by applets.
• Networking: URLs, TCP (Transmission Control Protocol), UDP (User Data gram Protocol) sockets, and IP (Internet Protocol) addresses.
• Internationalization: Help for writing programs that can be localized for users worldwide. Programs can automatically adapt to specific locales and be displayed in the appropriate language.
• Security: Both low level and high level, including electronic signatures, public and private key management, access control, and certificates.
• Software components: Known as JavaBeans^TM, can plug into existing component architectures.
• Object serialization: Allows lightweight persistence and communication via Remote Method Invocation (RMI).
• Java Database Connectivity (JDBC^TM): Provides uniform access to a wide range of relational databases.

The Java platform also has APIs for 2D and 3D graphics, accessibility, servers, collaboration, telephony, speech, animation, and more. The following figure depicts what is included in the Java 2 SDK.

 

6.4 HOW WILL JAVA TECHNOLOGY CHANGE MY LIFE?

We can’t promise you fame, fortune, or even a job if you learn the Java programming language. Still, it is likely to make your programs better and requires less effort than other languages. We believe that Java technology will help you do the following:

• Get started quickly: Although the Java programming language is a powerful object-oriented language, it’s easy to learn, especially for programmers already familiar with C or C++.
• Write less code: Comparisons of program metrics (class counts, method counts, and so on) suggest that a program written in the Java programming language can be four times smaller than the same program in C++.
• Write better code: The Java programming language encourages good coding practices, and its garbage collection helps you avoid memory leaks. Its object orientation, its JavaBeans component architecture, and its wide-ranging, easily extendible API let you reuse other people’s tested code and introduce fewer bugs.
• Develop programs more quickly: Your development time may be as much as twice as fast versus writing the same program in C++. Why? You write fewer lines of code and it is a simpler programming language than C++.
• Avoid platform dependencies with 100% Pure Java: You can keep your program portable by avoiding the use of libraries written in other languages. The 100% Pure Java^TM Product Certification Program has a repository of historical process manuals, white papers, brochures, and similar materials online.
• Write once, run anywhere: Because 100% Pure Java programs are compiled into machine-independent byte codes, they run consistently on any Java platform.
• Distribute software more easily: You can upgrade applets easily from a central server. Applets take advantage of the feature of allowing new classes to be loaded “on the fly,” without recompiling the entire program.

 

6.5 ODBC:

 

Microsoft Open Database Connectivity (ODBC) is a standard programming interface for application developers and database systems providers. Before ODBC became a de facto standard for Windows programs to interface with database systems, programmers had to use proprietary languages for each database they wanted to connect to. Now, ODBC has made the choice of the database system almost irrelevant from a coding perspective, which is as it should be. Application developers have much more important things to worry about than the syntax that is needed to port their program from one database to another when business needs suddenly change.

Through the ODBC Administrator in Control Panel, you can specify the particular database that is associated with a data source that an ODBC application program is written to use. Think of an ODBC data source as a door with a name on it. Each door will lead you to a particular database. For example, the data source named Sales Figures might be a SQL Server database, whereas the Accounts Payable data source could refer to an Access database. The physical database referred to by a data source can reside anywhere on the LAN.

The ODBC system files are not installed on your system by Windows 95. Rather, they are installed when you setup a separate database application, such as SQL Server Client or Visual Basic 4.0. When the ODBC icon is installed in Control Panel, it uses a file called ODBCINST.DLL. It is also possible to administer your ODBC data sources through a stand-alone program called ODBCADM.EXE. There is a 16-bit and a 32-bit version of this program and each maintains a separate list of ODBC data sources.

From a programming perspective, the beauty of ODBC is that the application can be written to use the same set of function calls to interface with any data source, regardless of the database vendor. The source code of the application doesn’t change whether it talks to Oracle or SQL Server. We only mention these two as an example. There are ODBC drivers available for several dozen popular database systems. Even Excel spreadsheets and plain text files can be turned into data sources. The operating system uses the Registry information written by ODBC Administrator to determine which low-level ODBC drivers are needed to talk to the data source (such as the interface to Oracle or SQL Server). The loading of the ODBC drivers is transparent to the ODBC application program. In a client/server environment, the ODBC API even handles many of the network issues for the application programmer.

The advantages of this scheme are so numerous that you are probably thinking there must be some catch. The only disadvantage of ODBC is that it isn’t as efficient as talking directly to the native database interface. ODBC has had many detractors make the charge that it is too slow. Microsoft has always claimed that the critical factor in performance is the quality of the driver software that is used. In our humble opinion, this is true. The availability of good ODBC drivers has improved a great deal recently. And anyway, the criticism about performance is somewhat analogous to those who said that compilers would never match the speed of pure assembly language. Maybe not, but the compiler (or ODBC) gives you the opportunity to write cleaner programs, which means you finish sooner. Meanwhile, computers get faster every year.

6.6 JDBC:

In an effort to set an independent database standard API for Java; Sun Microsystems developed Java Database Connectivity, or JDBC. JDBC offers a generic SQL database access mechanism that provides a consistent interface to a variety of RDBMSs. This consistent interface is achieved through the use of “plug-in” database connectivity modules, or drivers. If a database vendor wishes to have JDBC support, he or she must provide the driver for each platform that the database and Java run on.

To gain a wider acceptance of JDBC, Sun based JDBC’s framework on ODBC. As you discovered earlier in this chapter, ODBC has widespread support on a variety of platforms. Basing JDBC on ODBC will allow vendors to bring JDBC drivers to market much faster than developing a completely new connectivity solution.

JDBC was announced in March of 1996. It was released for a 90 day public review that ended June 8, 1996. Because of user input, the final JDBC v1.0 specification was released soon after.

The remainder of this section will cover enough information about JDBC for you to know what it is about and how to use it effectively. This is by no means a complete overview of JDBC. That would fill an entire book.

 

6.7 JDBC Goals:

Few software packages are designed without goals in mind. JDBC is one that, because of its many goals, drove the development of the API. These goals, in conjunction with early reviewer feedback, have finalized the JDBC class library into a solid framework for building database applications in Java.

The goals that were set for JDBC are important. They will give you some insight as to why certain classes and functionalities behave the way they do. The eight design goals for JDBC are as follows:

SQL Level API

The designers felt that their main goal was to define a SQL interface for Java. Although not the lowest database interface level possible, it is at a low enough level for higher-level tools and APIs to be created. Conversely, it is at a high enough level for application programmers to use it confidently. Attaining this goal allows for future tool vendors to “generate” JDBC code and to hide many of JDBC’s complexities from the end user.

SQL Conformance

SQL syntax varies as you move from database vendor to database vendor. In an effort to support a wide variety of vendors, JDBC will allow any query statement to be passed through it to the underlying database driver. This allows the connectivity module to handle non-standard functionality in a manner that is suitable for its users.

JDBC must be implemental on top of common database interfaces

The JDBC SQL API must “sit” on top of other common SQL level APIs. This goal allows JDBC to use existing ODBC level drivers by the use of a software interface. This interface would translate JDBC calls to ODBC and vice versa.

1. Provide a Java interface that is consistent with the rest of the Java system

Because of Java’s acceptance in the user community thus far, the designers feel that they should not stray from the current design of the core Java system.

• Keep it simple

This goal probably appears in all software design goal listings. JDBC is no exception. Sun felt that the design of JDBC should be very simple, allowing for only one method of completing a task per mechanism. Allowing duplicate functionality only serves to confuse the users of the API.

• Use strong, static typing wherever possible

Strong typing allows for more error checking to be done at compile time; also, less error appear at runtime.

• Keep the common cases simple

Because more often than not, the usual SQL calls used by the programmer are simple SELECT’s, INSERT’s, DELETE’s and UPDATE’s, these queries should be simple to perform with JDBC. However, more complex SQL statements should also be possible.

Finally we decided to precede the implementation using Java Networking.

And for dynamically updating the cache table we go for MS Access database.

Java ha two things: a programming language and a platform.

Java is a high-level programming language that is all of the following

Simple                                     Architecture-neutral

Object-oriented                       Portable

Distributed                              High-performance

Interpreted                              Multithreaded

Robust                                     Dynamic Secure

Java is also unusual in that each Java program is both compiled and interpreted. With a compile you translate a Java program into an intermediate language called Java byte codes the platform-independent code instruction is passed and run on the computer.

Compilation happens just once; interpretation occurs each time the program is executed. The figure illustrates how this works.

Java Program

Compilers

Interpreter

My Program

6.7 NETWORKING TCP/IP STACK:

The TCP/IP stack is shorter than the OSI one:

TCP is a connection-oriented protocol; UDP (User Datagram Protocol) is a connectionless protocol.

IP datagram’s:

The IP layer provides a connectionless and unreliable delivery system. It considers each datagram independently of the others. Any association between datagram must be supplied by the higher layers. The IP layer supplies a checksum that includes its own header. The header includes the source and destination addresses. The IP layer handles routing through an Internet. It is also responsible for breaking up large datagram into smaller ones for transmission and reassembling them at the other end.

UDP:

UDP is also connectionless and unreliable. What it adds to IP is a checksum for the contents of the datagram and port numbers. These are used to give a client/server model – see later.

TCP:

TCP supplies logic to give a reliable connection-oriented protocol above IP. It provides a virtual circuit that two processes can use to communicate.

Internet addresses

In order to use a service, you must be able to find it. The Internet uses an address scheme for machines so that they can be located. The address is a 32 bit integer which gives the IP address.

Network address:

Class A uses 8 bits for the network address with 24 bits left over for other addressing. Class B uses 16 bit network addressing. Class C uses 24 bit network addressing and class D uses all 32.

Subnet address:

Internally, the UNIX network is divided into sub networks. Building 11 is currently on one sub network and uses 10-bit addressing, allowing 1024 different hosts.

Host address:

8 bits are finally used for host addresses within our subnet. This places a limit of 256 machines that can be on the subnet.

Total address:

The 32 bit address is usually written as 4 integers separated by dots.

Port addresses

A service exists on a host, and is identified by its port. This is a 16 bit number. To send a message to a server, you send it to the port for that service of the host that it is running on. This is not location transparency! Certain of these ports are “well known”.

Sockets:

A socket is a data structure maintained by the system to handle network connections. A socket is created using the call socket. It returns an integer that is like a file descriptor. In fact, under Windows, this handle can be used with Read File and Write File functions.

#include <sys/types.h>

#include <sys/socket.h>

int socket(int family, int type, int protocol);

Here “family” will be AF_INET for IP communications, protocol will be zero, and type will depend on whether TCP or UDP is used. Two processes wishing to communicate over a network create a socket each. These are similar to two ends of a pipe – but the actual pipe does not yet exist.

6.8 JFREE CHART:

JFreeChart is a free 100% Java chart library that makes it easy for developers to display professional quality charts in their applications. JFreeChart’s extensive feature set includes:

A consistent and well-documented API, supporting a wide range of chart types;

A flexible design that is easy to extend, and targets both server-side and client-side applications;

Support for many output types, including Swing components, image files (including PNG and JPEG), and vector graphics file formats (including PDF, EPS and SVG);

JFreeChart is “open source” or, more specifically, free software. It is distributed under the terms of the GNU Lesser General Public Licence (LGPL), which permits use in proprietary applications.

 

6.8.1. Map Visualizations:

Charts showing values that relate to geographical areas. Some examples include: (a) population density in each state of the United States, (b) income per capita for each country in Europe, (c) life expectancy in each country of the world. The tasks in this project include: Sourcing freely redistributable vector outlines for the countries of the world, states/provinces in particular countries (USA in particular, but also other areas);

Creating an appropriate dataset interface (plus default implementation), a rendered, and integrating this with the existing XYPlot class in JFreeChart; Testing, documenting, testing some more, documenting some more.

6.8.2. Time Series Chart Interactivity

Implement a new (to JFreeChart) feature for interactive time series charts — to display a separate control that shows a small version of ALL the time series data, with a sliding “view” rectangle that allows you to select the subset of the time series data to display in the main chart.

6.8.3. Dashboards

There is currently a lot of interest in dashboard displays. Create a flexible dashboard mechanism that supports a subset of JFreeChart chart types (dials, pies, thermometers, bars, and lines/time series) that can be delivered easily via both Java Web Start and an applet.

 

6.8.4. Property Editors

The property editor mechanism in JFreeChart only handles a small subset of the properties that can be set for charts. Extend (or reimplement) this mechanism to provide greater end-user control over the appearance of the charts.

CHAPTER 7

7.0 APPENDIX

7.1 SAMPLE SCREEN SHOTS:

7.2 SAMPLE SOURCE CODE:

CHAPTER 8

8.1 CONCLUSION

In this paper, we have proposed architecture for the comprehensive IDS in AMI, which is designed to be reliable, dynamic, and considering the real-time nature of traffic for each component in AMI. Then, we conduct a performance analysis experiment of the seven existing state-of-the-art data stream mining algorithms on a public IDS data set. Finally, we elucidate the strengths and weaknesses of those algorithms and assess the suitability of each of them to serve as the IDSs for the three different components of AMI. We have observed that some algorithms that use very minimal amount of computing resources and offer moderate level of accuracy can potentially be used for the smart meter IDS. On the other hand, the algorithms that require more computing resources and offer higher accuracy levels can be useful for the IDSs in data concentrators and AMI headends.

8.2 FUTURE ENHANCEMENT:

As future work, we plan to develop our own lightweight yet accurate data stream mining algorithms to be used for the smart meter IDS and to set up a small-scale hardware platform for AMI to test our algorithms. In conclusion, we hope our research can make contributions toward more secure AMI deployments by the use of steam datamining-based IDSs.

This paper presents a systematic data-driven approach to assisting situational application development. We first propose a technique to extract useful information from multiple sources to abstract service capabilities with set tags. This supports intuitive expression of user’s desired composition goals by simple queries, without having to know underlying technical details. A planning technique then exploits composition solutions which can constitute the desired goals, even with some potential new interesting composition opportunities. A browser-based tool facilitates visual and iterative refinement of composition solutions, to finally come up with the satisfying outputs. A series of experiments demonstrate the efficiency and effectiveness of our approach. Data-driven composition technique for situational web applications by using tag-based semantics in to illustrate the overall life-cycle of our “compose as-you-search” composition approach, to propose the clustering technique for deriving tag-based composition semantics, and to evaluate the composition planning effectiveness, respectively.

Compared with previous work, this paper is significantly updated by introducing a semi-supervised technique for clustering hierarchical tag based semantics from service documentations and human-annotated annotations. The derived semantics link service capabilities and developers’ processing goals, so that the composition is processed by planning the “Tag HyperLinks” from initialquery to the goals. The planning algorithm is also further evaluated in terms of recommendation quality, performance, and scalability over data sets from real-world service repositories. Results show that our approach reaches satisfying precision and high-quality composition recommendations. We also demonstrate that our approach can accommodate even larger size of services than real world repositories so as to promise performance. Besides, more details of our interactive development prototyping are presented. We particularly demonstrate how the composition UI can help developers intuitively compose situational applications, and iteratively refine their goals until requirements are finally satisfied.

1.2 INTRODUCTION:

We develop and deliver software systems more quickly, and these systems must provide increasingly ambitious functionality to adapt ever-changing requirements and environments. Particularly, in recent a few years, the emergence and wide adoption of Web 2.0 have enlarged the body of service computing research. Web 2.0 not only focuses on the resource sharing and utilization from user and social perspective, but also exhibits the notion of “Web as a Platform” paradigm. A very important trend is that, more and more service consumers (including programmers, business analysts or even endusers) are capable of participating and collaborating for their own requirements and interests by means of developing situational software applications (also noted as “situated software”).

Software engineering perspective, situational software applications usually follow the opportunistic development fashion, where small subsets of users create applications to fulfill a specific purpose. Currently, composing available web-delivered services (including SOAP based web services, REST (RE presentational State Transfer) web services and RSS/Atom feeds) into a single web applications, or so called “service mashups” (or “mashups” for short) has been popular. They are supposed to be flexible response for new needs or demands and quick roll-out of some potentially unanticipated functionality. To support situational application development, a number of tools from both academia and industry have emerged.

However, we argue that, the large number of available services and the complexity of composition constraints make manual composition difficult. For the situational applications developers, who might be non-professional programmers, the key challenge remained is that they intend to represent their desired goals simply and intuitively, and be quickly navigated to proper service that can response their requests. They usually do not care about (or understand) the underlying technical details (e.g., syntactics, semantics, message mediation, etc). They just want to figure out all intermediate steps needed to generate desired outputs.

Moreover, many end-users may have a general wish to know what they are trying to achieve, but not know the specifics of what they want or what is possible. It means that the process of designing and developing the situational application requires not only the abstraction of individual services, but also much broader perspective on the evolving collections of services that can potentially incorporate with current onesWe first present a data-driven composition technique for situational web applications by using tag-based semantics in ICWS 2011 work.

The main contributions in this paper are to illustrate the overall life-cycle of our “composeas-you-search” composition approach, to propose the clustering technique for deriving tag-based composition semantics, and to evaluate the composition planning effectiveness, respectively. Compared with previous work, this paper is significantly updated by introducing a semi-supervised technique for clustering hierarchical tag-based semantics from service documentations and human-annotated annotations. The derived semantics link service capabilities and developers’ processing goals, so that the composition is processed by planning the “Tag HyperLinks” from initialquery to the goals.

The planning algorithm is also further evaluated in terms of recommendation quality, performance, and scalability over data sets from real-world service repositories. Results show that our approach reaches satisfying precision and high-quality composition recommendations. We also demonstrate that our approach can accommodate even larger size of services than real world repositories so as to promise performance. Besides, more details of our interactive development prototyping are presented. We particularly demonstrate how the composition UI can help developers intuitively compose situational applications, and iteratively refine their goals until requirements are finally satisfied.

1.3 SCOPE OF THE PROJECT

User-oriented abstraction: The tourist uses tags to represent their desired goals and find relevant services. Tags provide a uniform abstraction of user requirements and service capabilities, and lower the entry barrier to perform development. 

Data-driven development: In the whole development process, the tourist selects or inputs some tags, while some relevant services are recommended. This reflects a “Compose-as-you-Search” development process. Recommended services either process these tags as inputs, or produce these tags as outputs. As shown in Fig. 1, each service has some inputs and outputs, which are associated with tagged data. In this way, services can be connected to build data flows. Developers can search their goals by means of tags, and compose recommended services in a data driven fashion. 

Potential composition navigation: The developer is always assisted with possible composition suggestions, based on the tags in the current goals. Thecomposition engine interprets the user queries and automatically generates some appropriate compositions alternatives by a planning algorithm (Section 4). The recommendations not only contain the desired outputs from the developers’ goals, but also suggest some interesting or relevant suggestions leading to potential new composition possibilities.

For example, the tag “Italian” introduced the Google Translation service, which tourist was not aware of such composition possibility. In this way, the composition process is not like traditional semantic web services techniques which might need specific goals, but leads to some emergent opportunities according to current application situations.

1.4 LITRATURE SURVEY:

COMPOSING DATA-DRIVEN SERVICE MASHUPS WITH TAG-BASED SEMANTIC ANNOTATIONS

AUTHOR: X. Liu, Q. Zhao, G. Huang, H. Mei, and T. Teng

PUBLISH: Proc. IEEE Int’l Conf. Web Services (ICWS ’11), pp. 243-250, 2011.

EXPLANATION:

Spurred by Web 2.0 paradigm, there emerge large numbers of service mashups by composing readily accessible data and services. Mashups usually address solving situational problems and require quick and iterative development lifecyle. In this paper, we propose an approach to composing data driven mashups, based on tag-based semantics. The core principle is deriving semantic annotations from popular tags, and associating them with programmatic inputs and outputs data. Tag-based semantics promise a quick and simple comprehension of data capabilities. Mashup developers including end-users can intuitively search desired services with tags, and combine several services by means of data flows. Our approach takes a planning technique to retrieving the potentially relevant composition opportunities. With our graphical composition user interfaces, developers can iteratively modify, adjust and refine their mashups to be more satisfying.

TOWARDS AUTOMATIC TAGGING FOR WEB SERVICES

AUTHOR: L. Fang, L. Wang, M. Li, J. Zhao, Y. Zou, and L. Shao

PUBLISH: Proc. IEEE 19th Int’l Conf. Web Services, pp. 528-535, 2012.

EXPLANATION:

Tagging technique is widely used to annotate objects in Web 2.0 applications. Tags can support web service understanding, categorizing and discovering, which are important tasks in a service-oriented software system. However, most of existing web services’ tags are annotated manually. Manual tagging is time-consuming. In this paper, we propose a novel approach to tag web services automatically. Our approach consists of two tagging strategies, tag enriching and tag extraction. In the first strategy, we cluster web services using WSDL documents, and then we enrich tags for a service with the tags of other services in the same cluster. Considering our approach may not generate enough tags by tag enriching, we also extract tags from WSDL documents and related descriptions in the second step. To validate the effectiveness of our approach, a series of experiments are carried out based on web-scale web services. The experimental results show that our tagging method is effective, ensuring the number and quality of generated tags. We also show how to use tagging results to improve the performance of a web service search engine, which can prove that our work in this paper is useful and meaningful.

A TAG-BASED APPROACH FOR THE DESIGN AND COMPOSITION OF INFORMATION PROCESSING APPLICATIONS

AUTHOR: E. Bouillet, M. Feblowitz, Z. Liu, A. Ranganathan, and A. Riabov

PUBLISH: ACM SIGPLAN Notices, vol. 43, no. 10, pp. 585-602, Sept. 2008.

EXPLANATION:

In the realm of component-based software systems, pursuers of the holy grail of automated application composition face many significant challenges. In this paper we argue that, while the general problem of automated composition in response to high-level goal statements is indeed very difficult to solve, we can realize composition in a restricted context, supporting varying degrees of manual to automated assembly for specific types of applications. We propose a novel paradigm for composition in flow-based information processing systems, where application design and component development are facilitated by the pervasive use of faceted, tag-based descriptions of processing goals, of component capabilities, and of structural patterns of families of application. The facets and tags represent different dimensions of both data and processing, where each facet is modeled as a finite set of tags that are defined in a controlled folksonomy. All data flowing through the system, as well as the functional capabilities of components are described using tags. A customized AI planner is used to automatically build an application, in the form of a flow of components, given a high-level goal specification in the form of a set of tags. End-users use an automatically populated faceted search and navigation mechanism to construct these high-level goals. We also propose a novel software engineering methodology to design and develop a set of reusable, well-described components that can be assembled into a variety of applications. With examples from a case study in the Financial Services domain, we demonstrate that composition using a faceted, tag-based application design is not only possible, but also extremely useful in helping end-users create situational applications from a wide variety of available components.

CHAPTER 2

2.0 SYSTEM ANALYSIS

2.1 EXISTING SYSTEM:

In our previous work we have designed a technique to extract tags by mining service specifications (including WSDL, Web API documents and web pages that contain references to web services) and collecting human-generated contents (including comments and queries). Several web services tagging approaches have been proposed, for example the FCA tagging system most of them annotate web services manually. Manual tagging is a time consuming work. Moreover, several existing systems can recommend tags for web services based on existing handmade tags such as the approach these systems consider nothing about similarities between tags and web services. Another problem in these systems is that if there is no handmade tag, they cannot work at all. Another system CDKH in can generate tags for web services automatically, but the system doesn’t use existing handmade tags of web services.

These different ways are combined in tagging tools that the tag-based platform facilitates. Moreover, inside of the platform and due to the preferences of the users, different tagging behaviours exist that actually obstruct the automated interoperability among tag sets. Despite the fact that the systems offer solutions to aid the understanding of the folksonomy that the users collectively build (tag clouds, tools based on related tag ideas, collective intelligence methods, data mining, etc.) Although tagging shows potential benefits, personal organization of information leads to implicit logical conditions that often differ from the global one. Tagging provides a sort of weak organisation of the information, very useful, but mediated by the user’s behaviour. Therefore, it is also possible that user’s tags associated with an object do not agree with the other users tags.

2.1.1 DISADVANTAGES:

• There exist several limitations to collaborative tagging in sites such as Delicious. The first one is that a tag can be used to refer to different concepts; that is, there is a context dependent feature of the tag associated with the user.

• This dependence limits both the effectiveness and adequacy of collaborative tagging. The limitation is called”Context Dependent Knowledge Heterogeneity” (CDKH). A second is the Classical Ambiguity (CA) of terms, inherited from natural language and/or the consideration of different”basic levels” among users

• CA would not be critical when users work with urls (content of url induces, in fact, a disambiguation of terms because of its specific topic). In this case, the contextualization of tags in a graph structure (by means of clustering analysis) distinguishes the different terms associated with the same tag. CDKH is associated with concept structures that users do not represent in the system, but that FCA can extract.

2.2 PROPOSED SYSTEM:

We propose a heuristic graph-based planning algorithm within polynomial-time complexity. When the developer selects a tag from the tag cloud or input a keyword as the initial query request qi , the planning algorithm first computes the cost of achieving each tag starting from qi by conducting a forward search. Such a Depth-First Search step constructs all possible Tag Links that can perform the final goal. Based on the results above, the planning algorithm then approximates the sequence of Tag Links that connects qi to the final goal by a regression search step the tourist takes geographical locations of hotel, restaurant, bars and museum, we cannot give the reasonable order for visiting these places. Preferences, quality, ordering and other constraints would be helpful to improve the plan quality and performance. Due to the popularity and simplicity of tags, our tag-based service model can be extended, where all these constraints can be also presented as tags.   

Our approach relies on the popularity of tags on the web. The primitive of tag-based composition of flow applications was first proposed in the MARIO system. Tag-based search is a hot topic in the research body of information retrieval and data mining. Most of existing research works focus on processing tags from popular social networking sites like Del.icio.us, Twitter and flickr. To best of our knowledge, few works have been made in the area of existing service-based applications. The primitive of tag-based composition of flow applications was first proposed in the MARIO system. Some recent works try to leverage tag-based service discovery, but not fully consider the hierarchy relationships of tags.

Our approach provides a systematic way for extracting useful tags from service documents and user generated annotations, by fully considering the unique features of web services like interface naming rules and developer preferences. Besides traditional similarity-based measurement, the clustering process is also controlled by the probability of tag occurrence and its own property, without any needs of training data. It should be noted that, we currently make simple mapping between our top-level tags to WordNet. However, the search results seem to be satisfying in regular cases.

2.2.1 ADVANTAGES:

• Tag extraction and clustering: Tags are extracted from multiple sources, including service textual documentation, user-generated comments and queries, etc. Browsing such a large size of tags is really tedious, and tag ambiguity might cause mistakes. Therefore, a semi-supervised technique is proposed to cluster tag-based taxonomy as uni- fied semantic foundation.

• Composition semantics derivation: Service providers and application architects are responsible of annotating tagbased semantics to describe service capabilities, including functionalities, input and output data, and other useful information. Based on the generated tag hierarchy, some rules can help them accomplish the semantic annotation semi-automatically.

• Composition goal search: In our browser-based development environment, developers can search their desired goals using tag queries. As tags are easier and more intuitive to understand, developers only focus on their desired goals without having to know underlying technical information of services. The queries are immediately submitted to the composition engine.

• Composition planning: A composition engine interprets tag queries, and generates appropriate solutions that can contain or accomplish the goal. Our composition engine employs a graph-based planning technique to generate possible composition recommendations. As discussed above, this process retrieves prefabricated composition logics from task templates, or generates potentially new alternatives. Recommendations might be either individual services, or a set of services connected by data flows.
  

2.3 HARDWARE & SOFTWARE REQUIREMENTS:

2.3.1 HARDWARE REQUIREMENT:

v    Processor                                 –    Pentium –IV

• Speed                                      –    1.1 GHz
  • RAM                                       –    256 MB (min)
  • Hard Disk                               –   20 GB
  • Floppy Drive                           –    1.44 MB
  • Key Board                              –    Standard Windows Keyboard
  • Mouse                                     –    Two or Three Button Mouse
  • Monitor                                   –    SVGA

 

2.3.2 SOFTWARE REQUIREMENTS:

• Operating System                   :           Windows XP or Win7
• Front End                                :           Microsoft Visual Studio 2008
• Back End                                :           MSSQL Server
• Server                                      :           ASP Sever Page
• Script                                       :           C# Script
• Document                               :           MS-Office 2007

CHAPTER 3

3.0 SYSTEM DESIGN:

Data Flow Diagram / Use Case Diagram / Flow Diagram:

• The DFD is also called as bubble chart. It is a simple graphical formalism that can be used to represent a system in terms of the input data to the system, various processing carried out on these data, and the output data is generated by the system

• The data flow diagram (DFD) is one of the most important modeling tools. It is used to model the system components. These components are the system process, the data used by the process, an external entity that interacts with the system and the information flows in the system.

• DFD shows how the information moves through the system and how it is modified by a series of transformations. It is a graphical technique that depicts information flow and the transformations that are applied as data moves from input to output.

• DFD is also known as bubble chart. A DFD may be used to represent a system at any level of abstraction. DFD may be partitioned into levels that represent increasing information flow and functional detail.

NOTATION:

SOURCE OR DESTINATION OF DATA:

External sources or destinations, which may be people or organizations or other entities

DATA SOURCE:

Here the data referenced by a process is stored and retrieved.

PROCESS:

People, procedures or devices that produce data’s in the physical component is not identified.

DATA FLOW:

Data moves in a specific direction from an origin to a destination. The data flow is a “packet” of data.

MODELING RULES:

There are several common modeling rules when creating DFDs:

1. All processes must have at least one data flow in and one data flow out.
2. All processes should modify the incoming data, producing new forms of outgoing data.
3. Each data store must be involved with at least one data flow.
4. Each external entity must be involved with at least one data flow.
5. A data flow must be attached to at least one process.

3.1 ARCHITECTURE DIAGRAM

3.2 DATAFLOW DIAGRAM

ADMIN:

USER:

UML DIAGRAMS:

3.2 USE CASE DIAGRAM:

ADMIN:

USER:

3.3 CLASS DIAGRAM:

ADMIN:

USER:

3.4 SEQUENCE DIAGRAM:

ADMIN:

USER:

3.5 ACTIVITY DIAGRAM:

ADMIN:

USER:

CHAPTER 4

4.0 IMPLEMENTATION:

MARIO SYSTEM:

MARIO system is the most prior work to leverage tagbased descriptions as component annotations, whereby users can find desired goals by regular search. Based on the SPPL planner MARIO facilitates combination of components to create applications that satisfy end-user goals. Our approach shares common insights and learns successful experiences of MARIO. However, MARIO holds two assumptions: the tag-based semantics have to be prede- fined, and (2) the tag-based descriptions of all components might be (manually) pre-solved. The assumptions are quite reasonable to deal with relatively smaller component repository size or in a specific application domain with controlled vocabulary.

However, problems are yet remained in real-world scenarios: most of currently available web services and mashups are not with enough meaningful tags. In well-known repositories like ProgrammableWeb, 1 Seekda, 2 and Service-Finder, 3 existing tags are too limited and trivial to determine composition. For example, these tags can mainly help service categorization (like travel, education, games), but not provide sufficient enough information to reveal the relationships among services. In a sense, collecting enough tags and deriving semantics between them are indispensable step to achieve automated composition. Moreover, the quality of derived semantics should also be evaluated.

we have designed a technique to extract tags by mining service specifications (including WSDL, Web API documents and web pages that contain references to web services) and collecting human-generated contents (including comments and queries). Our work provides a similarity-based measurements including structure metric, lexical metric and frequency metric. We have obtained a repository with the size of 50,000 tags, which were extracted from over 20,000 real-world web services and 6,000 mashups. Initial experiments also showed that the tag-based search could improve the search performance and quality of a single web service. Based on the collected tags, this paper particularly addresses following three issues: (1) how to abstract tags for simple and precise service discovery; (2) how to identify the potential composition of a set of services by their tag-based descriptions; (3) how to operate the composition efficiently, even with the large size of tags.

4.1 ALGORITHM:

TAG CLUSTERING WITH ANNEALING ALGORITHM:

We apply a semi-supervised model to derive hierarchical structure from tags T annotating the services. It begins with the root node containing all tags in T and recursively splits them into a series of semantically meaningful clusters. The process does not terminate until each cluster represents a specific concept. At the final step of this algorithm, a cluster usually corresponds to a high-level category of a set of tags. For example, the tag set containing {country, street, city, milan, Italy, zipcode} represents the concept “geography”, and the one containing {rain, sunny, windchill, 27C, 80 F} is associated with the concept “weather”.

Our approach tries to generate a “Feature Tag” ô to summarize the semantics of other tags in the cluster (like “weather” in the example above), in order to navigate to high-level compositional semantics close to their desired goals. We briefly illustrate the splitting process as follows. At the beginning, we maintain a queue Q to store the information of all the nodes that are waiting for splitting, and a vector n in the queue indicates the probability that each tag emerges in this node. Initially all elements of n0 are assigned to 1, for all tags that are contained in the root node. In the clustering process, an annealing algorithm is employed to split the tags into several semantically meaningful clusters. Such optimizing algorithm could be stated as a process minimizing a predefined criterion.

4.2 MODULES:

DATA SET PREPARATION:

TAG-BASED SERVICE MODEL:

SEMANTICS DERIVATION:

TAG IDENTIFICATION AND EXTRACTION:

DATA-DRIVEN COMPOSITION:

4.3 MODULE DESCRIPTION:

DATA SET PREPARATION:

Tag semantics play the crucial role in our composition approach. We build up our service community in the Trustie Project,4 which is a testbed environment for software service production. The platform crawls web services and mashups from some well-known repositories like ProgrammableWeb and Seekda. As our approach takes input/output as composition unit, each API and each operation in WSDL is stored as one item. The data set in this experiment includes 19,083 service items. These items were put into some categories with statistics, like travel (728), news (484), weather (1,491), maps (792), geography (1,822), food (273), photo (489), messaging (816), blogging (332), and so on. Some sample services with their tags could be found via our website. We first applied the splitting technique to extract tags from textual descriptions. Then we manually filtered the redundant tags. For example, the three tags “Map”, “Maps” and “Mapping” are considered as one tag. Finally we chose a data set of 23,971 different tags. Applying algorithm for tag clustering and the EM processing, we attained 594 clusters such as hotel, geography, weather, search, map, etc.

TAG-BASED SERVICE MODEL:

Our data-driven, goal-oriented composition technique in the key primitive is the tag-based data flow between services. There are two constraints in terms of service composition: syntactic and semantic. In our approach, semantic constraints can be inferred from the hierarchical tag semantics; syntactic constraints depend on our underlying composition middleware, which takes responsibility of dealing with actual data types required and produced by the service. Based on the tag-based semantics, if a web service ws1 can produce t1 as its output, and the service ws2 can consume t1 or its father tag t2 as its input, we consider that ws1 and ws2 can be composed, since a data flow can be created between them. From this perspective, the tag-based service composition problem is defined as the result of creating a data flow of a sequence of tags. Just like the hyperlinks form the navigation among web pages, we call the tag-based data flow using the notion of Tag Link (TL) in the following. The first precondition indicates the mapping and propagation between web services at semantic level, which relies on the derived tag-based hierarchy. The second precondition ensures that no extra data is left at the syntactic level. Note that all selected services in the Tag-Links will be encapsulated according to our iMashup component model and composition runtime takes charge of interpreting and coordinating underlying technical details such as data object types and structures.

SEMANTICS DERIVATION:

Generally, efficient service composition relies on the precision of candidate services that are discovered. The precision of candidate services search also reflects the quality of derived tag-based semantics. Hereby, we tried to evaluate the precision of our tag-based search. We compared the results with traditional Term Frequency-Inverse Document Frequency (TF/IDF) retrieval technique for searching a single web service. We computed the similarities between the input query and web services in formula 14 by referencing classic formula defined by Manning.

Our planning approach is the discovery of potential composition opportunities. According to common experiences, the number of candidate services often implies the number of concepts. As about 90 percent mashups on Programmable Web contain less than five services, to make a comparison baseline, we chose 20 sample applications, each of which at least contains eight services. For each sample application, we still employed the same 20 junior students to manually extract the tags of the services, or add new annotations based from tag-based taxonomy. For each output and the user inputs, we ran the planning composition algorithm that might retrieve the same output given the user inputs and form an application incrementally. We compared the planned solutions with the original applications.

TAG IDENTIFICATION AND EXTRACTION:

Tags are actually a set of keywords to describe some aspects of a service. We extract tags from two main sources: (1) Service textual descriptions; (2) User-generated annotations. We briefly illustrate how to process them for extracting tags. For textual descriptions like WSDL documents, tags can be extracted from elements including SERVICES, INTERFACE, MESSAGE, TYPE and DOCUMENTATION. Usually, useful tags reside in: (1) service name containing the general information; (2) service interfaces describing service usage (including operations and input/output messages). From our investigation, we observe that over 90 percent WSDL documents use capital letters, numbers, or “ ” to separate tokens in service names [16]. So we use the following rules to split tokens into tags.  Capital letters, numbers, “%” and “ ” are treated as starting position of a new word.  First position is also treated as a starting position of a new word.  Contiguous, single capital letters and numbers should be merged into one token. For example, according to our rules, we split service name “AmazonSimpleDB” into {Amazon, Simple, DB}. For service interfaces, they are usually defined in form of “verb” plus “noun” e.g., “postZipcodeRequest”, “getHotelInfoResponse”, “getCompanyInfoResponse”. Verbs can reflect the type of messages: “post” and “request” are usually used for input messages, while “get” and “return” are usually used for output messages. In contrast, nouns may reflect more plentiful usage information of the services. So we extract the nouns and ignore verbs. For example, consider the output message “getHotelInfoResponse”, useful tags are {hotelname, address, zipcode, telephone, tax}.

DATA-DRIVEN COMPOSITION:

Our development process can be generally described as following steps in Fig. 2: Tag extraction and clustering. Tags are extracted from multiple sources, including service textual documentation, user-generated comments and queries, etc. (step ❶). Browsing such a large size of tags is really tedious, and tag ambiguity might cause mistakes. Therefore, a semi-supervised technique is proposed to cluster tag-based taxonomy as uni- fied semantic foundation (step ❷). Composition semantics derivation. Service providers and application architects are responsible of annotating tagbased semantics to describe service capabilities, including functionalities, input and output data, and other useful information. Based on the generated tag hierarchy, some rules can help them accomplish the semantic annotation semi-automatically. This (step ❸) aims to make services compatibly composed by the Tag-Link model (Section 5.1). Composition goal search. In our browser-based development environment, developers can search their desired goals using tag queries (step ❹). As tags are easier and more intuitive to understand, developers only focus on their desired goals without having to know underlying technical information of services. The queries are immediately submitted to the composition engine. Composition planning. A composition engine interprets tag queries, and generates appropriate solutions that can contain or accomplish the goal (step ❺). Our composition engine employs a graph-based planning technique to generate possible composition recommendations. As discussed above, this process retrieves prefabricated composition logics from task templates, or generates potentially new alternatives. Recommendations might be either individual services, or a set of services connected by data flows. Composition visualization, refinement and refactoring. Developers are able to directly run the generated compositions within a browser-based environment. At each composition step, the developers can revise immediate composition results, and iteratively refractor or re-design composition results (steps ❻ and ❼). Such a feature makes developers know what exactly the current composition results are. In this way, they can visually and iteratively refine composition requirements, until the final outputs are satisfied. In other words, steps ❹ to ❼ are iteratively performed.

CHAPTER 5

5.0 SYSTEM STUDY:

5.1 FEASIBILITY STUDY:

The feasibility of the project is analyzed in this phase and business proposal is put forth with a very general plan for the project and some cost estimates. During system analysis the feasibility study of the proposed system is to be carried out. This is to ensure that the proposed system is not a burden to the company.  For feasibility analysis, some understanding of the major requirements for the system is essential.

Three key considerations involved in the feasibility analysis are 

• ECONOMICAL FEASIBILITY
• TECHNICAL FEASIBILITY
• SOCIAL FEASIBILITY

5.1.1 ECONOMICAL FEASIBILITY:     

This study is carried out to check the economic impact that the system will have on the organization. The amount of fund that the company can pour into the research and development of the system is limited. The expenditures must be justified. Thus the developed system as well within the budget and this was achieved because most of the technologies used are freely available. Only the customized products had to be purchased.

 

5.1.2 TECHNICAL FEASIBILITY   

This study is carried out to check the technical feasibility, that is, the technical requirements of the system. Any system developed must not have a high demand on the available technical resources. This will lead to high demands on the available technical resources. This will lead to high demands being placed on the client. The developed system must have a modest requirement, as only minimal or null changes are required for implementing this system.  

5.1.3 SOCIAL FEASIBILITY:  

The aspect of study is to check the level of acceptance of the system by the user. This includes the process of training the user to use the system efficiently. The user must not feel threatened by the system, instead must accept it as a necessity. The level of acceptance by the users solely depends on the methods that are employed to educate the user about the system and to make him familiar with it. His level of confidence must be raised so that he is also able to make some constructive criticism, which is welcomed, as he is the final user of the system.

5.2 SYSTEM TESTING:

Testing is a process of checking whether the developed system is working according to the original objectives and requirements. It is a set of activities that can be planned in advance and conducted systematically. Testing is vital to the success of the system. System testing makes a logical assumption that if all the parts of the system are correct, the global will be successfully achieved. In adequate testing if not testing leads to errors that may not appear even many months.

This creates two problems, the time lag between the cause and the appearance of the problem and the effect of the system errors on the files and records within the system. A small system error can conceivably explode into a much larger Problem. Effective testing early in the purpose translates directly into long term cost savings from a reduced number of errors. Another reason for system testing is its utility, as a user-oriented vehicle before implementation. The best programs are worthless if it produces the correct outputs.

5.2.1 UNIT TESTING:

Description	Expected result
Test for application window properties.	All the properties of the windows are to be properly aligned and displayed.
Test for mouse operations.	All the mouse operations like click, drag, etc. must perform the necessary operations without any exceptions.

A program represents the logical elements of a system. For a program to run satisfactorily, it must compile and test data correctly and tie in properly with other programs. Achieving an error free program is the responsibility of the programmer. Program  testing  checks  for  two  types  of  errors:  syntax  and  logical. Syntax error is a program statement that violates one or more rules of the language in which it is written. An improperly defined field dimension or omitted keywords are common syntax errors. These errors are shown through error message generated by the computer. For Logic errors the programmer must examine the output carefully.

5.1.2 FUNCTIONAL TESTING:

Functional testing of an application is used to prove the application delivers correct results, using enough inputs to give an adequate level of confidence that will work correctly for all sets of inputs. The functional testing will need to prove that the application works for each client type and that personalization function work correctly.When a program is tested, the actual output is compared with the expected output. When there is a discrepancy the sequence of instructions must be traced to determine the problem.  The process is facilitated by breaking the program into self-contained portions, each of which can be checked at certain key points. The idea is to compare program values against desk-calculated values to isolate the problems.

Description	Expected result
Test for all modules.	All peers should communicate in the group.
Test for various peer in a distributed network framework as it display all users available in the group.	The result after execution should give the accurate result.

5.1. 3 NON-FUNCTIONAL TESTING:

 The Non Functional software testing encompasses a rich spectrum of testing strategies, describing the expected results for every test case. It uses symbolic analysis techniques. This testing used to check that an application will work in the operational environment. Non-functional testing includes:

• Load testing
• Performance testing
• Usability testing
• Reliability testing
• Security testing

5.1.4 LOAD TESTING:

An important tool for implementing system tests is a Load generator. A Load generator is essential for testing quality requirements such as performance and stress. A load can be a real load, that is, the system can be put under test to real usage by having actual telephone users connected to it. They will generate test input data for system test.

Description	Expected result
It is necessary to ascertain that the application behaves correctly under loads when ‘Server busy’ response is received.	Should designate another active node as a Server.

5.1.5 PERFORMANCE TESTING:

Performance tests are utilized in order to determine the widely defined performance of the software system such as execution time associated with various parts of the code, response time and device utilization. The intent of this testing is to identify weak points of the software system and quantify its shortcomings.

Description	Expected result
This is required to assure that an application perforce adequately, having the capability to handle many peers, delivering its results in expected time and using an acceptable level of resource and it is an aspect of operational management.	Should handle large input values, and produce accurate result in a  expected time.

5.1.6 RELIABILITY TESTING:

The software reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time and it is being ensured in this testing. Reliability can be expressed as the ability of the software to reveal defects under testing conditions, according to the specified requirements. It the portability that a software system will operate without failure under given conditions for a given time interval and it focuses on the behavior of the software element. It forms a part of the software quality control team.

Description	Expected result
This is to check that the server is rugged and reliable and can handle the failure of any of the components involved in provide the application.	In case of failure of  the server an alternate server should take over the job.

5.1.7 SECURITY TESTING:

Security testing evaluates system characteristics that relate to the availability, integrity and confidentiality of the system data and services. Users/Clients should be encouraged to make sure their security needs are very clearly known at requirements time, so that the security issues can be addressed by the designers and testers.

Description	Expected result
Checking that the user identification is authenticated.	In case failure it should not be connected in the framework.
Check whether group keys in a tree are shared by all peers.	The peers should know group key in the same group.

5.1.8 WHITE BOX TESTING:

White  box  testing,  sometimes called  glass-box  testing is  a test  case  design method  that  uses  the  control  structure  of the procedural  design  to  derive  test  cases. Using  white  box  testing  method,  the software  engineer  can  derive  test  cases. The White box testing focuses on the inner structure of the software structure to be tested.

Description	Expected result
Exercise all logical decisions on their true and false sides.	All the logical decisions must be valid.
Execute all loops at their boundaries and within their operational bounds.	All the loops must be finite.
Exercise internal data structures to ensure their validity.	All the data structures must be valid.

5.1.9 BLACK BOX TESTING:

Black box testing, also called behavioral testing, focuses on the functional requirements of the software.  That  is,  black  testing  enables  the software engineer  to  derive  sets  of  input  conditions  that  will  fully  exercise  all  functional requirements  for  a  program.  Black box testing is not alternative to white box techniques.  Rather  it  is  a  complementary  approach  that  is  likely  to  uncover  a different  class  of  errors  than  white box  methods. Black box testing attempts to find errors which focuses on inputs, outputs, and principle function of a software module. The starting point of the black box testing is either a specification or code. The contents of the box are hidden and the stimulated software should produce the desired results.

Description	Expected result
To check for incorrect or missing functions.	All the functions must be valid.
To check for interface errors.	The entire interface must function normally.
To check for errors in a data structures or external data base access.	The database updation and retrieval must be done.
To check for initialization and termination errors.	All the functions and data structures must be initialized properly and terminated normally.

All the above system testing strategies are carried out in as the development, documentation and institutionalization of the proposed goals and related policies is essential.

CHAPTER 6

6.0 SOFTWARE DESCRIPTION:

 

6.1 JAVA TECHNOLOGY:

Java technology is both a programming language and a platform.

 

The Java Programming Language

 

The Java programming language is a high-level language that can be characterized by all of the following buzzwords:

• Simple
  • Architecture neutral
  • Object oriented
  • Portable
  • Distributed     
  • High performance
  • Interpreted     
  • Multithreaded
  • Robust
  • Dynamic
  • Secure     

With most programming languages, you either compile or interpret a program so that you can run it on your computer. The Java programming language is unusual in that a program is both compiled and interpreted. With the compiler, first you translate a program into an intermediate language called Java byte codes —the platform-independent codes interpreted by the interpreter on the Java platform. The interpreter parses and runs each Java byte code instruction on the computer. Compilation happens just once; interpretation occurs each time the program is executed. The following figure illustrates how this works.

You can think of Java byte codes as the machine code instructions for the Java Virtual Machine (Java VM). Every Java interpreter, whether it’s a development tool or a Web browser that can run applets, is an implementation of the Java VM. Java byte codes help make “write once, run anywhere” possible. You can compile your program into byte codes on any platform that has a Java compiler. The byte codes can then be run on any implementation of the Java VM. That means that as long as a computer has a Java VM, the same program written in the Java programming language can run on Windows 2000, a Solaris workstation, or on an iMac.

6.2 THE JAVA PLATFORM:

A platform is the hardware or software environment in which a program runs. We’ve already mentioned some of the most popular platforms like Windows 2000, Linux, Solaris, and MacOS. Most platforms can be described as a combination of the operating system and hardware. The Java platform differs from most other platforms in that it’s a software-only platform that runs on top of other hardware-based platforms.

The Java platform has two components:

• The Java Virtual Machine (Java VM)
• The Java Application Programming Interface (Java API)

You’ve already been introduced to the Java VM. It’s the base for the Java platform and is ported onto various hardware-based platforms.

The Java API is a large collection of ready-made software components that provide many useful capabilities, such as graphical user interface (GUI) widgets. The Java API is grouped into libraries of related classes and interfaces; these libraries are known as packages. The next section, What Can Java Technology Do? Highlights what functionality some of the packages in the Java API provide.

The following figure depicts a program that’s running on the Java platform. As the figure shows, the Java API and the virtual machine insulate the program from the hardware.

Native code is code that after you compile it, the compiled code runs on a specific hardware platform. As a platform-independent environment, the Java platform can be a bit slower than native code. However, smart compilers, well-tuned interpreters, and just-in-time byte code compilers can bring performance close to that of native code without threatening portability.

6.3 WHAT CAN JAVA TECHNOLOGY DO?

The most common types of programs written in the Java programming language are applets and applications. If you’ve surfed the Web, you’re probably already familiar with applets. An applet is a program that adheres to certain conventions that allow it to run within a Java-enabled browser.

However, the Java programming language is not just for writing cute, entertaining applets for the Web. The general-purpose, high-level Java programming language is also a powerful software platform. Using the generous API, you can write many types of programs.

An application is a standalone program that runs directly on the Java platform. A special kind of application known as a server serves and supports clients on a network. Examples of servers are Web servers, proxy servers, mail servers, and print servers. Another specialized program is a servlet.

A servlet can almost be thought of as an applet that runs on the server side. Java Servlets are a popular choice for building interactive web applications, replacing the use of CGI scripts. Servlets are similar to applets in that they are runtime extensions of applications. Instead of working in browsers, though, servlets run within Java Web servers, configuring or tailoring the server.

How does the API support all these kinds of programs? It does so with packages of software components that provides a wide range of functionality. Every full implementation of the Java platform gives you the following features:

• The essentials: Objects, strings, threads, numbers, input and output, data structures, system properties, date and time, and so on.
• Applets: The set of conventions used by applets.
• Networking: URLs, TCP (Transmission Control Protocol), UDP (User Data gram Protocol) sockets, and IP (Internet Protocol) addresses.
• Internationalization: Help for writing programs that can be localized for users worldwide. Programs can automatically adapt to specific locales and be displayed in the appropriate language.
• Security: Both low level and high level, including electronic signatures, public and private key management, access control, and certificates.
• Software components: Known as JavaBeans^TM, can plug into existing component architectures.
• Object serialization: Allows lightweight persistence and communication via Remote Method Invocation (RMI).
• Java Database Connectivity (JDBC^TM): Provides uniform access to a wide range of relational databases.

The Java platform also has APIs for 2D and 3D graphics, accessibility, servers, collaboration, telephony, speech, animation, and more. The following figure depicts what is included in the Java 2 SDK.

 

6.4 HOW WILL JAVA TECHNOLOGY CHANGE MY LIFE?

We can’t promise you fame, fortune, or even a job if you learn the Java programming language. Still, it is likely to make your programs better and requires less effort than other languages. We believe that Java technology will help you do the following:

• Get started quickly: Although the Java programming language is a powerful object-oriented language, it’s easy to learn, especially for programmers already familiar with C or C++.
• Write less code: Comparisons of program metrics (class counts, method counts, and so on) suggest that a program written in the Java programming language can be four times smaller than the same program in C++.
• Write better code: The Java programming language encourages good coding practices, and its garbage collection helps you avoid memory leaks. Its object orientation, its JavaBeans component architecture, and its wide-ranging, easily extendible API let you reuse other people’s tested code and introduce fewer bugs.
• Develop programs more quickly: Your development time may be as much as twice as fast versus writing the same program in C++. Why? You write fewer lines of code and it is a simpler programming language than C++.
• Avoid platform dependencies with 100% Pure Java: You can keep your program portable by avoiding the use of libraries written in other languages. The 100% Pure Java^TM Product Certification Program has a repository of historical process manuals, white papers, brochures, and similar materials online.
• Write once, run anywhere: Because 100% Pure Java programs are compiled into machine-independent byte codes, they run consistently on any Java platform.
• Distribute software more easily: You can upgrade applets easily from a central server. Applets take advantage of the feature of allowing new classes to be loaded “on the fly,” without recompiling the entire program.

 

6.5 ODBC:

 

Microsoft Open Database Connectivity (ODBC) is a standard programming interface for application developers and database systems providers. Before ODBC became a de facto standard for Windows programs to interface with database systems, programmers had to use proprietary languages for each database they wanted to connect to. Now, ODBC has made the choice of the database system almost irrelevant from a coding perspective, which is as it should be. Application developers have much more important things to worry about than the syntax that is needed to port their program from one database to another when business needs suddenly change.

Through the ODBC Administrator in Control Panel, you can specify the particular database that is associated with a data source that an ODBC application program is written to use. Think of an ODBC data source as a door with a name on it. Each door will lead you to a particular database. For example, the data source named Sales Figures might be a SQL Server database, whereas the Accounts Payable data source could refer to an Access database. The physical database referred to by a data source can reside anywhere on the LAN.

The ODBC system files are not installed on your system by Windows 95. Rather, they are installed when you setup a separate database application, such as SQL Server Client or Visual Basic 4.0. When the ODBC icon is installed in Control Panel, it uses a file called ODBCINST.DLL. It is also possible to administer your ODBC data sources through a stand-alone program called ODBCADM.EXE. There is a 16-bit and a 32-bit version of this program and each maintains a separate list of ODBC data sources.

From a programming perspective, the beauty of ODBC is that the application can be written to use the same set of function calls to interface with any data source, regardless of the database vendor. The source code of the application doesn’t change whether it talks to Oracle or SQL Server. We only mention these two as an example. There are ODBC drivers available for several dozen popular database systems. Even Excel spreadsheets and plain text files can be turned into data sources. The operating system uses the Registry information written by ODBC Administrator to determine which low-level ODBC drivers are needed to talk to the data source (such as the interface to Oracle or SQL Server). The loading of the ODBC drivers is transparent to the ODBC application program. In a client/server environment, the ODBC API even handles many of the network issues for the application programmer.

The advantages of this scheme are so numerous that you are probably thinking there must be some catch. The only disadvantage of ODBC is that it isn’t as efficient as talking directly to the native database interface. ODBC has had many detractors make the charge that it is too slow. Microsoft has always claimed that the critical factor in performance is the quality of the driver software that is used. In our humble opinion, this is true. The availability of good ODBC drivers has improved a great deal recently. And anyway, the criticism about performance is somewhat analogous to those who said that compilers would never match the speed of pure assembly language. Maybe not, but the compiler (or ODBC) gives you the opportunity to write cleaner programs, which means you finish sooner. Meanwhile, computers get faster every year.

6.6 JDBC:

In an effort to set an independent database standard API for Java; Sun Microsystems developed Java Database Connectivity, or JDBC. JDBC offers a generic SQL database access mechanism that provides a consistent interface to a variety of RDBMSs. This consistent interface is achieved through the use of “plug-in” database connectivity modules, or drivers. If a database vendor wishes to have JDBC support, he or she must provide the driver for each platform that the database and Java run on.

To gain a wider acceptance of JDBC, Sun based JDBC’s framework on ODBC. As you discovered earlier in this chapter, ODBC has widespread support on a variety of platforms. Basing JDBC on ODBC will allow vendors to bring JDBC drivers to market much faster than developing a completely new connectivity solution.

JDBC was announced in March of 1996. It was released for a 90 day public review that ended June 8, 1996. Because of user input, the final JDBC v1.0 specification was released soon after.

The remainder of this section will cover enough information about JDBC for you to know what it is about and how to use it effectively. This is by no means a complete overview of JDBC. That would fill an entire book.

 

6.7 JDBC Goals:

Few software packages are designed without goals in mind. JDBC is one that, because of its many goals, drove the development of the API. These goals, in conjunction with early reviewer feedback, have finalized the JDBC class library into a solid framework for building database applications in Java.

The goals that were set for JDBC are important. They will give you some insight as to why certain classes and functionalities behave the way they do. The eight design goals for JDBC are as follows:

SQL Level API

The designers felt that their main goal was to define a SQL interface for Java. Although not the lowest database interface level possible, it is at a low enough level for higher-level tools and APIs to be created. Conversely, it is at a high enough level for application programmers to use it confidently. Attaining this goal allows for future tool vendors to “generate” JDBC code and to hide many of JDBC’s complexities from the end user.

SQL Conformance

SQL syntax varies as you move from database vendor to database vendor. In an effort to support a wide variety of vendors, JDBC will allow any query statement to be passed through it to the underlying database driver. This allows the connectivity module to handle non-standard functionality in a manner that is suitable for its users.

JDBC must be implemental on top of common database interfaces

The JDBC SQL API must “sit” on top of other common SQL level APIs. This goal allows JDBC to use existing ODBC level drivers by the use of a software interface. This interface would translate JDBC calls to ODBC and vice versa.

1. Provide a Java interface that is consistent with the rest of the Java system

Because of Java’s acceptance in the user community thus far, the designers feel that they should not stray from the current design of the core Java system.

• Keep it simple

This goal probably appears in all software design goal listings. JDBC is no exception. Sun felt that the design of JDBC should be very simple, allowing for only one method of completing a task per mechanism. Allowing duplicate functionality only serves to confuse the users of the API.

• Use strong, static typing wherever possible

Strong typing allows for more error checking to be done at compile time; also, less error appear at runtime.

• Keep the common cases simple

Because more often than not, the usual SQL calls used by the programmer are simple SELECT’s, INSERT’s, DELETE’s and UPDATE’s, these queries should be simple to perform with JDBC. However, more complex SQL statements should also be possible.

Finally we decided to precede the implementation using Java Networking.

And for dynamically updating the cache table we go for MS Access database.

Java ha two things: a programming language and a platform.

Java is a high-level programming language that is all of the following

Simple                                     Architecture-neutral

Object-oriented                       Portable

Distributed                              High-performance

Interpreted                              Multithreaded

Robust                                     Dynamic Secure

Java is also unusual in that each Java program is both compiled and interpreted. With a compile you translate a Java program into an intermediate language called Java byte codes the platform-independent code instruction is passed and run on the computer.

Compilation happens just once; interpretation occurs each time the program is executed. The figure illustrates how this works.

6.7 NETWORKING TCP/IP STACK:

The TCP/IP stack is shorter than the OSI one:

TCP is a connection-oriented protocol; UDP (User Datagram Protocol) is a connectionless protocol.

IP datagram’s:

The IP layer provides a connectionless and unreliable delivery system. It considers each datagram independently of the others. Any association between datagram must be supplied by the higher layers. The IP layer supplies a checksum that includes its own header. The header includes the source and destination addresses. The IP layer handles routing through an Internet. It is also responsible for breaking up large datagram into smaller ones for transmission and reassembling them at the other end.

UDP:

UDP is also connectionless and unreliable. What it adds to IP is a checksum for the contents of the datagram and port numbers. These are used to give a client/server model – see later.

TCP:

TCP supplies logic to give a reliable connection-oriented protocol above IP. It provides a virtual circuit that two processes can use to communicate.

Internet addresses

In order to use a service, you must be able to find it. The Internet uses an address scheme for machines so that they can be located. The address is a 32 bit integer which gives the IP address.

Network address:

Class A uses 8 bits for the network address with 24 bits left over for other addressing. Class B uses 16 bit network addressing. Class C uses 24 bit network addressing and class D uses all 32.

Subnet address:

Internally, the UNIX network is divided into sub networks. Building 11 is currently on one sub network and uses 10-bit addressing, allowing 1024 different hosts.

Host address:

8 bits are finally used for host addresses within our subnet. This places a limit of 256 machines that can be on the subnet.

Total address:

The 32 bit address is usually written as 4 integers separated by dots.

Port addresses

A service exists on a host, and is identified by its port. This is a 16 bit number. To send a message to a server, you send it to the port for that service of the host that it is running on. This is not location transparency! Certain of these ports are “well known”.

Sockets:

A socket is a data structure maintained by the system to handle network connections. A socket is created using the call socket. It returns an integer that is like a file descriptor. In fact, under Windows, this handle can be used with Read File and Write File functions.

#include <sys/types.h>

#include <sys/socket.h>

int socket(int family, int type, int protocol);

Here “family” will be AF_INET for IP communications, protocol will be zero, and type will depend on whether TCP or UDP is used. Two processes wishing to communicate over a network create a socket each. These are similar to two ends of a pipe – but the actual pipe does not yet exist.

6.8 JFREE CHART:

JFreeChart is a free 100% Java chart library that makes it easy for developers to display professional quality charts in their applications. JFreeChart’s extensive feature set includes:

A consistent and well-documented API, supporting a wide range of chart types;

A flexible design that is easy to extend, and targets both server-side and client-side applications;

Support for many output types, including Swing components, image files (including PNG and JPEG), and vector graphics file formats (including PDF, EPS and SVG);

JFreeChart is “open source” or, more specifically, free software. It is distributed under the terms of the GNU Lesser General Public Licence (LGPL), which permits use in proprietary applications.

 

6.8.1. Map Visualizations:

Charts showing values that relate to geographical areas. Some examples include: (a) population density in each state of the United States, (b) income per capita for each country in Europe, (c) life expectancy in each country of the world. The tasks in this project include: Sourcing freely redistributable vector outlines for the countries of the world, states/provinces in particular countries (USA in particular, but also other areas);

Creating an appropriate dataset interface (plus default implementation), a rendered, and integrating this with the existing XYPlot class in JFreeChart; Testing, documenting, testing some more, documenting some more.

6.8.2. Time Series Chart Interactivity

Implement a new (to JFreeChart) feature for interactive time series charts — to display a separate control that shows a small version of ALL the time series data, with a sliding “view” rectangle that allows you to select the subset of the time series data to display in the main chart.

6.8.3. Dashboards

There is currently a lot of interest in dashboard displays. Create a flexible dashboard mechanism that supports a subset of JFreeChart chart types (dials, pies, thermometers, bars, and lines/time series) that can be delivered easily via both Java Web Start and an applet.

 

6.8.4. Property Editors

The property editor mechanism in JFreeChart only handles a small subset of the properties that can be set for charts. Extend (or reimplement) this mechanism to provide greater end-user control over the appearance of the charts.

CHAPTER 7

7.0 APPENDIX

7.1 SAMPLE SCREEN SHOTS:

7.2 SAMPLE SOURCE CODE:

CHAPTER 8

8.1 CONCLUSION

This paper presents our experiences of flowbased mashup development and tooling. The key principle of our approach is “Compose-as-you-Search”, which leverages tag-based service composition to lower the entry barrier for mashup development, and realizes the philosophy of “live development” by providing on-the-fly recommendations as well as the visual iterative refinement of applications. The key limitations of current approach are also addressed. As most service-oriented situational software applications are not appropriate for classic enterprise software that has strict quality requirements for security, availability, or performance. The approach proposed in this paper mainly targets at personal and small-scale data processing problems.

8.2 FUTURE ENHANCEMENT: One of our future directions is to accumulate useful composition knowledge. Composition knowledge retrieval is a recently hot topic in mashup development. We will attempt to combine our work with Knowledge Discovery from Service (KDS) proposed to extend the tag-based model to be more expressive beyond functionality specifications. For example, we can annotate a constraint by “tag “to the driving guide, to plan visiting orders. Certainly, the planning algorithm should add these constraints for scheduling actions.