Veleco

A Distributed Three-Hop Routing Protocolto Increase the Capacity of HybridWireless NetworksHaiying Shen, Senior Member, IEEE, Ze Li, and Chenxi QiuAbstract—Hybrid wireless networks combining the advantages of both mobile ad-hoc networks and infrastructure wireless networkshave been receiving increased attention due to their ultra-high performance. An efficient data routing protocol is important in suchnetworks for high network capacity and scalability. However, most routing protocols for these networks simply combine the ad-hoctransmission mode with the cellular transmission mode, which inherits the drawbacks of ad-hoc transmission. This paper presents aDistributed Three-hop Routing protocol (DTR) for hybrid wireless networks. To take full advantage of the widespread base stations,DTR divides a message data stream into segments and transmits the segments in a distributed manner. It makes full spatial reuse of asystem via its high speed ad-hoc interface and alleviates mobile gateway congestion via its cellular interface. Furthermore, sendingsegments to a number of base stations simultaneously increases throughput and makes full use of widespread base stations. Inaddition, DTR significantly reduces overhead due to short path lengths and the elimination of route discovery and maintenance. DTRalso has a congestion control algorithm to avoid overloading base stations. Theoretical analysis and simulation results show thesuperiority of DTR in comparison with other routing protocols in terms of throughput capacity, scalability, and mobility resilience. Theresults also show the effectiveness of the congestion control algorithm in balancing the load between base stations.Index Terms—Hybrid wireless networks, routing algorithm, load balancing, congestion controlÇ1 INTRODUCTIONOVER the past few years, wireless networks includinginfrastructure wireless networks and mobile ad-hocnetworks (MANETs) have attracted significant researchinterest. The growing desire to increase wireless networkcapacity for high performance applications has stimulatedthe development of hybrid wireless networks [1], [2], [3],[4], [5], [6]. A hybrid wireless network consists of both aninfrastructure wireless network and a mobile ad-hoc network.Wireless devices such as smart-phones, tablets andlaptops, have both an infrastructure interface and an ad-hocinterface. As the number of such devices has been increasingsharply in recent years, a hybrid transmission structurewill be widely used in the near future. Such a structure synergisticallycombines the inherent advantages and overcomethe disadvantages of the infrastructure wirelessnetworks and mobile ad-hoc networks.In a mobile ad-hoc network, with the absence of a centralcontrol infrastructure, data is routed to its destinationthrough the intermediate nodes in a multi-hop manner. Themulti-hop routing needs on-demand route discovery orroute maintenance [7], [8], [9], [10]. Since the messages aretransmitted in wireless channels and through dynamic routingpaths, mobile ad-hoc networks are not as reliable asinfrastructure wireless networks. Furthermore, because ofthe multi-hop transmission feature, mobile ad-hoc networksare only suitable for local area data transmission.The infrastructure wireless network (e.g., cellular network)is the major means of wireless communication in ourdaily lives. It excels at inter-cell communication (i.e., communicationbetween nodes in different cells) and Internetaccess. It makes possible the support of universal networkconnectivity and ubiquitous computing by integrating allkinds of wireless devices into the network. In an infrastructurenetwork, nodes communicate with each other throughbase stations (BSes). Because of the long distance one-hoptransmission between BSes and mobile nodes, the infrastructurewireless networks can provide higher messagetransmission reliability and channel access efficiency, butsuffer from higher power consumption on mobile nodesand the single point of failure problem [11].A hybrid wireless network synergistically combines aninfrastructure wireless network and a mobile ad-hoc networkto leverage their advantages and overcome theirshortcomings, and finally increases the throughput capacityof a wide-area wireless network. A routing protocol is a criticalcomponent that affects the throughput capacity of awireless network in data transmission. Most current routingprotocols in hybrid wireless networks [1], [5], [6], [12], [13],[14], [15], [16], [17], [18] simply combine the cellular transmissionmode (i.e., BS transmission mode) in infrastructurewireless networks and the ad-hoc transmission mode inmobile ad-hoc networks [7], [8], [9]. That is, as shown inFig. 1a, the protocols use the multi-hop routing to forward amessage to the mobile gateway nodes that are closest to theBSes or have the highest bandwidth to the BSes. The bandwidthof a channel is the maximum throughput (i.e.,_ The authors are with the Department of Electrical and ComputerEngineering, Clemson University, Clemson, SC 29634.E-mail: {shenh, zel, chenxiq}@clemson.edu.Manuscript received 18 Mar. 2014; accepted 18 Dec. 2014. Date of publication7 Jan. 2015; date of current version 31 Aug. 2015.For information on obtaining reprints of this article, please send e-mail to:reprints@ieee.org, and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TMC.2015.2388476IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015 19751536-1233 _ 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.transmission rate in bits/s) that can be achieved. The mobilegateway nodes then forward the messages to the BSes, functioningas bridges to connect the ad-hoc network and theinfrastructure network.However, direct combination of the two transmissionmodes inherits the following problems that are rooted inthe ad-hoc transmission mode._ High overhead. Route discovery and maintenance incurhigh overhead. The wireless random access mediumaccess control (MAC) required in mobile ad-hoc networks,which utilizes control handshaking and aback-off mechanism, further increases overhead._ Hot spots. The mobile gateway nodes can easilybecome hot spots. The RTS-CTS random access, inwhich most traffic goes through the same gateway,and the flooding employed in mobile ad-hoc routingto discover routes may exacerbate the hot spotproblem. In addition, mobile nodes only use thechannel resources in their route direction, whichmay generate hot spots while leave resources inother directions under-utilized. Hot spots lead tolow transmission rates, severe network congestion,and high data dropping rates._ Low reliability. Dynamic and long routing paths leadto unreliable routing. Noise interference and neighborinterference during the multi-hop transmissionprocess cause a high data drop rate. Long routingpaths increase the probability of the occurrence ofpath breakdown due to the highly dynamic natureof wireless ad-hoc networks.These problems become an obstacle in achieving highthroughput capacity and scalability in hybrid wireless networks.Considering the widespread BSes, the mobile nodeshave a high probability of encountering a BS while moving.Taking advantage of this feature, we propose a DistributedThree-hop Data Routing protocol (DTR). In DTR, as shownin Fig. 1b, a source node divides a message stream into anumber of segments. Each segment is sent to a neighbormobile node. Based on the QoS requirement, these mobilerelay nodes choose between direct transmission or relaytransmission to the BS. In relay transmission, a segment isforwarded to another mobile node with higher capacity to aBS than the current node. In direct transmission, a segmentis directly forwarded to a BS. In the infrastructure, the segmentsare rearranged in their original order and sent to thedestination. The number of routing hops in DTR is confinedto three, including at most two hops in the ad-hoc transmissionmode and one hop in the cellular transmission mode.To overcome the aforementioned shortcomings, DTR triesto limit the number of hops. The first hop forwarding distributesthe segments of a message in different directions tofully utilize the resources, and the possible second hop forwardingensures the high capacity of the forwarder. DTRalso has a congestion control algorithm to balance the trafficload between the nearby BSes in order to avoid traffic congestionat BSes.Using self-adaptive and distributed routing with highspeedand short-path ad-hoc transmission, DTR significantlyincreases the throughput capacity and scalability ofhybrid wireless networks by overcoming the three shortcomingsof the previous routing algorithms. It has the followingfeatures:_ Low overhead. It eliminates overhead caused by routediscovery and maintenance in the ad-hoc transmissionmode, especially in a dynamic environment._ Hot spot reduction. It alleviates traffic congestion atmobile gateway nodes while makes full use of channelresources through a distributed multi-path relay._ High reliability. Because of its small hop path lengthwith a short physical distance in each step, it alleviatesnoise and neighbor interference and avoids theadverse effect of route breakdown during data transmission.Thus, it reduces the packet drop rate andmakes full use of spacial reuse, in which severalsource and destination nodes can communicatesimultaneously without interference.The rest of this paper is organized as follows. Section 2presents a review of representative hybrid wireless networksand multi-hop routing protocols. Section 3 details theDTR protocol, with an emphasis on its routing methods,segment structure, and BS congestion control. Section 4 theoreticallyanalyzes the performance of the DTR protocol.Section 5 shows the performance of the DTR protocol incomparison to other routing protocols. Finally, Section 6concludes the paper.2 RELATED WORKIn order to increase the capacity of hybrid wireless networks,various routing methods with different featureshave been proposed. One group of routing methods integratethe ad-hoc transmission mode and the cellular transmissionmode [1], [5], [6], [14], [16], [17], [18]. Dousse et al.[6] built a Poisson Boolean model to study how a BSincreases the capacity of a MANET. Lin and Hsu [5] proposeda Multihop Cellular Network (MCN) and derived itsthroughput. Hsieh and Sivakumar [14] investigated ahybrid IEEE 802.11 network architecture with both a distributedcoordination function and a point coordination function.Luo et al. [1] proposed a unified cellular and ad-hocnetwork architecture for wireless communication. Cho andHaas [16] studied the impact of concurrent transmission ina downlink direction (i.e., from BSes to mobile nodes) onthe system capacity of a hybrid wireless network. In [17],[18], a node initially communicates with other nodes usingan ad-hoc transmission mode, and switches to a cellularFig. 1. Traditional and proposed routing algorithms on the uplinkdirection.1976 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015transmission mode when its performance is better than thead-hoc transmission.The above methods are only used to assist intra-cell adhoctransmission rather than inter-cell transmission. Ininter-cell transmission [1], [5], [6], a message is forwardedvia the ad-hoc interface to the gateway mobile node that isclosest to or has the highest uplink transmission bandwidthto a BS. The gateway mobile node then forwards the messageto the BS using the cellular interface. However, most ofthese routing protocols simply combine routing schemes inad-hoc networks and infrastructure networks, hence inheritthe drawbacks of the ad-hoc transmission mode asexplained previously.DTR is similar to the Two-hop transmission protocol [19]in terms of the elimination of route maintenance and thelimited number of hops in routing. In Two-hop, when anode’s bandwidth to a BS is larger than that of each neighbor,it directly sends a message to the BS. Otherwise, itchooses a neighbor with a higher channel and sends a messageto it, which further forwards the message to the BS.DTR is different from Two-hop in three aspects. First, Twohoponly considers the node transmission within a singlecell, while DTR can also deal with inter-cell transmission,which is more challenging and more common than intracellcommunication in the real world. Second, DTR uses distributedtransmission involving multiple cells, which makesfull use of system resources and dynamically balances thetraffic load between neighboring cells. In contrast, Two-hopemploys single-path transmission.There are other methods proposed to improve routingperformance in hybrid wireless networks. Wu et al. [3]proposed using ad-hoc relay stations to dynamically relaytraffic from one cell to another in order to avoid traffic congestionin BSes. Li et al. [20] surveyed a number of multihopcellular network architectures in literature, andcompared and discussed methods to reduce the cost ofdeployment for MCNs. The work in [21] investigates howto allocate the bandwidth to users to improve the performanceof hybrid wireless networks. Thulasiraman andShen [22] further considered the wireless interference inoptimizing the resource allocation in hybrid wireless networks.The work in [23] proposes a coalitional game theorybased cooperative packet delivery scheme in hybridwireless networks. There are also some works [24], [25],[26] that study radio frequency allocation for directiontransmission and relay transmission in hybrid wirelessnetworks. These works are orthogonal to our study in thispaper and can be incorporated into DTR to furtherenhance its performance.The throughput capacity of the hybrid wireless networkunder different settings has also been an activeresearch topic in the hybrid wireless network. The worksin [17], [27] have studied the throughput of hybrid networkwith n nodes and m stations. Liu et al. [28] theoreticallystudied the capacity of hybrid wireless networksunder an one-dimensional network topology and a twodimensionalstrip topology. Wang et al. [29] studied themulticast throughput of hybrid wireless networks anddesigned an optimal multicast strategy based on deducedthroughput.3 DISTRIBUTED THREE-HOP ROUTING PROTOCOL3.1 Assumption and OverviewSince BSes are connected with a wired backbone, we assumethat there are no bandwidth and power constraints on transmissionsbetween BSes. We use intermediate nodes to denoterelay nodes that function as gateways connecting an infrastructurewireless network and a mobile ad-hoc network.We assume every mobile node is dual-mode; that is, it hasad-hoc network interface such as a WLAN radio interfaceand infrastructure network interface such as a 3G cellularinterface.DTR aims to shift the routing burden from the ad-hocnetwork to the infrastructure network by taking advantageof widespread base stations in a hybrid wireless network.Rather than using one multi-hop path to forward a messageto one BS, DTR uses at most two hops to relay the segmentsof a message to different BSes in a distributed manner, andrelies on BSes to combine the segments. Fig. 2 demonstratesthe process of DTR in a hybrid wireless network. We simplifythe routings in the infrastructure network for clarity.As shown in the figure, when a source node wants to transmita message stream to a destination node, it divides themessage stream into a number of partial streams called segmentsand transmits each segment to a neighbor node.Upon receiving a segment from the source node, a neighbornode locally decides between direct transmission andrelay transmission based on the QoS requirement of theapplication. The neighbor nodes forward these segments ina distributed manner to nearby BSes. Relying on the infrastructurenetwork routing, the BSes further transmit thesegments to the BS where the destination node resides. Thefinal BS rearranges the segments into the original order andforwards the segments to the destination. It uses the cellularIP transmission method [30] to send segments to thedestination if the destination moves to another BS duringsegment transmission.Our DTR algorithm avoids the shortcomings of ad-hoctransmission in the previous routing algorithms thatdirectly combine an ad-hoc transmission mode and a cellulartransmission mode. Rather than using the multi-hop adhoctransmission, DTR uses two hop forwarding by relyingon node movement and widespread base stations. All otheraspects remain the same as those in the previous routingalgorithms (including the interaction with the TCP layer).DTR works on the Internet layer. It receives packets fromFig. 2. Data transmission in the DTR protocol.SHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1977the TCP layer and routes it to the destination node, whereDTR forwards the packet to the TCP layer.The data routing process in DTR can be divided intotwo steps: uplink from a source node to the first BS anddownlink from the final BS to the data’s destination. Criticalproblems that need to be solved include how a sourcenode or relay node chooses nodes for efficient segmentforwarding, and how to ensure that the final BS sendssegments in the right order so that a destination nodereceives the correct data. Also, since traffic is not evenlydistributed in the network, how to avoid overloadingBSes is another problem. Below, Section 3.2 will presentthe details for forwarding node selection in uplink transmissionand Section 3.3 will present the segment structurethat helps ensure the correct final order of segments in amessage, and DTR’s strategy for downlink transmission.Section 3.4 will present the congestion control algorithmfor balancing a load between BSes.3.2 Uplink Data RoutingA long routing path will lead to high overhead, hot spotsand low reliability. Thus, DTR tries to limit the path length.It uses one hop to forward the segments of a message in adistributed manner and uses another hop to find highcapacityforwarder for high performance routing. As aresult, DTR limits the path length of uplink routing to twohops in order to avoid the problems of long-path multi-hoprouting in the ad-hoc networks. Specifically, in the uplinkrouting, a source node initially divides its message streaminto a number of segments, then transmits the segments toits neighbor nodes. The neighbor nodes forward segmentsto BSes, which will forward the segments to the BS wherethe destination resides.Below, we first explain how to definecapacity, then introduce the way for a node to collect thecapacity information from its neighbors, and finally presentthe details of the DTR routing algorithm.Different applications may have different QoS requirements,such as efficiency, throughput, and routing speed.For example, delay-tolerant applications (e.g., voice mail,e-mail and text messaging) do not necessarily need fastreal-time transmission and may make throughput thehighest consideration to ensure successful data transmission.Some applications may take high mobility as theirpriority to avoid hot spots and blank spots. Hot spots areareas where BS channels are congested, while blank spotsare areas without signals or with very weak signals. Highmobilitynodes can quickly move out of a hot spot orblank spot and enter a cell with high bandwidth to a BS,thus providing efficient data transmission. Throughputcan be measured by bandwidth, mobility can be measuredby the speed of node movement, and routing speed can bemeasured by the speed of data forwarding. Bandwidthcan be estimated using the non-intrusive technique proposedin [31]. In this work, we take throughput and routingspeed as examples for the QoS requirement. We use abandwidth/queue metric to reflect node capacity in throughputand fast data forwarding. The metric is the ratio of anode’s channel bandwidth to its message queue size. Alarger bandwidth/queue value means higher throughputand message forwarding speed, and vice versa.When choosing neighbors for data forwarding, a nodeneeds the capacity information (i.e., queue size and bandwidth)of its neighbors. Also, a selected neighbor shouldhave enough storage space for a segment. To keep track ofthe capacity and storage space of its neighbors, each nodeperiodically exchanges its current capacity and storageinformation with its neighbors. In the ad-hoc network component,every node needs to periodically send “hello” messagesto identify its neighbors. Taking advantage of thispolicy, nodes piggyback the capacity and storage informationonto the “hello” messages in order to reduce the overheadcaused by the information exchanges. If a node’scapacity and storage space are changed after its last “hello”message sending when it receives a segment, it sends itscurrent capacity and storage information to the segment forwarder.Then, the segment forwarder will choose the highestcapacity nodes in its neighbors based on the mostupdated information.When a source node sends out message segments, itchooses the neighbors that have enough space for storing asegment, and then chooses neighbors that have the highestcapacity. In order to find higher capacity forwarders in alarger neighborhood around the source, each segmentreceiver further forwards its received segment to its neighborwith the highest capacity. That is, after a neighbor nodemi receives a segment from the source, it uses either directtransmission or relay transmission. If the capacity of each ofits neighbors is no greater than itself, relay node mi usesdirect transmission. Otherwise, it uses relay transmission.In direct transmission, the relay node sends the segment toa BS if it is in a BS’s region. Otherwise, it stores the segmentwhile moving until it enters a BS’s region. In relay transmission,relay node mi chooses its highest-capacity neighbor asthe second relay node based on the QoS requirement. Thesecond relay node will use direct transmission to forwardthe segment directly to a BS. As a result, the number oftransmission hops in the ad-hoc network component is confinedto no more than two. The small number of hops helpto increase the capacity of the network and reduce channelcontention in ad-hoc transmission. Algorithm 1 shows thepseudo-code for neighbor node selection and message forwardingin DTR.The purpose of the second hop selection is to find ahigher capacity node as the message forwarder in order toimprove the performance of the QoS requirement. As theneighborhood scope of a node for high-capacity nodesearching grows, the probability of finding higher capacitynodes increases. Thus, a source node’s neighbors are morelikely to find neighbors with higher capacities than thesource node. Therefore, transmitting data segments toneighbors and enabling them to choose the second relayshelp to find higher capacity nodes to forward data. If asource node has the highest capacity in its region, the segmentswill be forwarded back to the source node accordingto the DTR protocol. The source node then forwards the segmentsto the BSes directly due to the three-hop limit.Though sending data back and forth leads to latency andbandwidth wastage, this case occurs only when the sourcenodes is the highest capacity node within its two-hop neighborhood.Also, this step is necessary for finding the highestcapacity nodes within the source’s two-hop neighborhood,1978 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015and ensures that the highest capacity nodes are alwaysselected as the message forwarders. If the source node doesnot distribute segments to its neighbors, the higher-capacitynode searching cannot be conducted. Note that the datatransmission rate of the ad-hoc interface (e.g., IEEE 802.11)is more than 10 times faster than the cellular interface (e.g.,GSM, 3G). Thus, the transmission delay for sending thedata back and forth in the ad-hoc transmission is negligiblein the total routing latency.Algorithm 1. Pseudo-Code for Neighbor Node Selectionand Message Forwarding.1: ChooseRelay( ) f2: //choose neighbors with sufficient caches and bandwidth/queue(b/q) rates3: Query storage size and QoS requirement info. fromneighbors4: for each neighbor n do5: if n.cache.size>segment.length&&n.b/q>this.b/q then6: Add n toR ¼ fr1; . . . rm; . . .g in a descending order ofb/q7: end if8: end for9: Return R10: g11: Transmission( ) f12: if it is a source node then13: //routing conducted by a source node14: //choose relay nodes based on QoS requirement15: R¼ChooseRelay( );16: Send segments to fr1; . . . rmg in R17: else18: //routing conducted by a neighbor node19: if this.b/q _ b/q of all neighbors then20: //direct transmission21: if within the range of a BS then22: Transmit the segment directly to the BS23: end if24: else25: //relay transmission26: nodei ¼ getHighestCapability(ChooseRelay( ))27: Send a segment to nodei28: end if29: end if30: gBy distributing a message’s segments to different nodesto be forwarded in different directions, our algorithmreduces the congestion in the previous routing algorithmsin the hybrid wireless networks. When a node selects a relayto forward a segment, it checks the capacity of the node.Only when a node, say node mi, has enough capacity, thenode will forward a segment to node mi. Therefore, eventhough the paths are not node-disjoint, there will be no congestionin the common sub-paths.Fig. 3 shows examples of neighbor selection in DTR, inwhich the source node is in the transmission range of a BS.In the figures, the value in the node represents its capacity.In scenario (a), there exist nodes that have higher capacitythan the source node within the source’s two-hop neighborhood.If a routing algorithm directly let a source node transmita message to its BS, the high routing performancecannot be guaranteed since the source node may have verylow capacity. In DTR, the source node sends segments to itsneighbors, which further forward the segments to nodeswith higher capacities. In scenario (b), the source node hasthe highest capacity among the nodes in its two-hop neighborhood.After receiving segments from the source node,some neighbors forward the segments back to the sourcenode, which sends the message to its BS. Thus, DTR alwaysarranges data to be forwarded by nodes with high capacityto their BSes. DTR achieves higher throughput and fasterdata forwarding speed by taking into account node capacityin data forwarding.3.3 Downlink Data Routing and DataReconstructionAs mentioned above, the message stream of a source nodeis divided into several segments. After a BS receives a segment,it needs to forward the segment to the BS, where thedestination node resides (i.e., the destination BS). We usethe mobile IP protocol [32] to enable BSes to know thedestination BS. In this protocol, each mobile node is associatedwith a home BS, which is the BS in the node’s homenetwork, regardless of its current location in the network.The home network of a node contains its registrationinformation identified by its home address, which is astatic IP address assigned by an ISP. In a hybrid wirelessnetwork, each BS periodically emits beacon signals tolocate the mobile nodes in its range. When a mobile nodemi moves away from its home BS, the BS where mi currentlyresides detects mi and sends its IP address to thehome BS of mi. When a BS wants to contact mi, it contactsthe home BS of mi to find the destination BS where mi currentlyresides at.However, the destination BS recorded in the home BSmay not be the most up-to-date destination BS since destinationmobile nodes switch between the coverage regions ofdifferent BSes during data transmission to them. Forinstance, data is transmitted to BS Bi that has the data’s destination,but the destination has moved to the range of BSBj before the data arrives at BS Bi. To deal with this problem,we adopt the Cellular IP protocol [30] for trackingnode locations. With this protocol, a BS has a home agentand a foreign agent. The foreign agent keeps track of mobilenodes moving into the ranges of other BSes. The home agentintercepts in-coming segments, reconstructs the originaldata, and re-routes it to the foreign agent, which then forwardsthe data to the destination mobile node.Fig. 3. Neighbor selection in DTR.SHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1979After the destination BS receives the segments of a message,it rearranges the segments into the original messageand then sends it to the destination mobile node. A vitalissue is guaranteeing that the segments are combined in thecorrect order. For this purpose, DTR specifies the segmentstructure format. Each segment contains eight fields, including:(1) source node IP address (denoted by S); (2) destinationnode IP address (denoted by D); (3) message sequencenumber (denoted by m); (4) segment sequence number(denoted by s); (5) QoS indication number (denoted by q);(6) data; (7) length of the data; and (8) checksum. Fields (1)-(5) are in the segment head.The role of the source IP address field is to inform the destinationnode where the message comes from. Thedestination IP address field indicates the destination node,and is used to locate the final BS. After sending out a messagestream to a destination, a source node may send outanother message stream to the same destination node. Themessage sequence number differentiates the different messagestreams initiated by the same source node. The segmentsequence number is used to find the correct transmissionsequence of the segments for transmission to a destinationnode. The data is the actual information that a source nodewants to transmit to a destination node. The length fieldspecifies the length of the DTR segment including theheader in bytes. The checksum is used by the receiver nodeto check whether the received data has errors. The QoS indicationnumber is used to indicate the QoS requirement of theapplication.Thus, each segment’s head includes the informationrepresented by ðS; D; m; s; qÞðm; s ¼ 1; 2; 3; . . .Þ. When asegment with head ðS; D; m; s; qÞ arrives at a BS, the BScontacts D’s home BS to find the destination BS where Dstays via the mobile IP protocol. It then transmits the segmentto the destination BS through the infrastructure networkcomponent. After arriving at the BS, the segmentwaits in the cache for its turn to be transmitted to its destinationnode based on its message and segment sequencenumbers. At this time, if another segment comes with ahead labelled ðS; D; ðm þ 1Þ; s; qÞ, which means that it isfrom the same source node but belongs to another datastream, the BS will put it to another stream. If the segmentis labeled as ðS; D;m; ðs þ 1Þ; qÞ, it means that this segmentbelongs to the same data stream of the same sourcenode as segment ðS;D; m; s; qÞ. The combination of thesource node’s sequence number and segment sequencenumber helps to locate the stream and the position of asegment in the steam. In order to integrate the segmentsinto their correct order to retrieve the original data, thesegments in the BS are transmitted to the destinationnode in the order of the segments’ sequence in the originalmessage. If a segment has not arrived at the final BS,its subsequent segments will wait in the final BS until itsarrival. Algorithm 2 shows the pseudo-code for a BS toreorder and forward segments to their destinations. Notethat in the cache, we can set the timer based on the packetrate and storage limit. In other words, the timer should beset as large as possible to fully utilize the storage on BSesto ensure that a message has a high probability to berecovered.Algorithm 2. Pseudo-Code for a BS to Reorder andForward Segments to Destination Nodes.1: //a cache pool is built for each data stream2: //there are n cache pools currently3: if receives a segment (S, D, m, s, q) then4: if there is no cache pool with msg sequence num equalsm then5: Create a cache pool n þ 1 for the stream m6: else7: //the last delivered segment of stream m has sequence numi _ 18: if s ¼¼ i then9: Send out segment (S, D, m, s, q) to D10: iþþ;11: else12: Add segment (S, D, m, s) into cache pool m13: end if14: end if15: end if3.4 Congestion Control in Base StationsCompared to the previous routing algorithms in hybridwireless networks, DTR can distribute traffic load amongmobile nodes more evenly. Though the distributed routingin DTR can distribute traffic load among nearby BSes, if thetraffic load is not distributed evenly in the network, someBSes may become overloaded while other BSes remainlightly loaded. We propose a congestion control algorithmto avoid overloading BSes in uplink transmission (e.g., B1,B2 and B3 in Fig. 1b) and downlink transmission (e.g., B4 inFig. 1b), respectively.In the hybrid wireless network, BSes send beacon messagesto identify nearby mobile nodes. Taking advantage ofthis beacon strategy, once the workload of a BS, say Bi,exceeds a pre-defined threshold, Bi adds an extra bit in itsbeacon message to broadcast to all the nodes in its transmissionrange. Then, nodes near Bi know that Bi is overloadedand will not forward segments to Bi. When a node near Bi,say mi, needs to forward a segment to a BS, it will send thesegment to Bi based on the DTR algorithm. In our congestioncontrol algorithm, because Bi is overloaded, ratherthan targeting Bi, mi will forward the segment to a lightlyloaded neighboring BS of Bi. To this end, node mi firstqueries a multi-hop path to a lightly loaded neighboring BSof Bi. Node mi broadcasts a query message into the system.We set the TTL for the path query forwarding step to a constant(e.g., 3). The query message is forwarded along othernodes until a node (say mj) near a lightly loaded BS (say Bj)is reached. Due to broadcasting, a node may receive multiplecopies of the same queries. Each node only remembersmi and the node that forwards the first query (i.e., its precedingnode), and ignores all other the same queries. In thisway, a multi-hop path between the source node and thelightly loaded base station can be formed. Nodemj responds to the path query by adding a reply bit and theaddress of mi into its beacon message to its preceding nodein the path. This beacon receiver also adds a reply bit andthe address of mi into its beacon message to its precedingnode in the path. This process repeats until mi receives thebeacon. Thus, each node knows its preceding node and1980 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015succeeding node in the path from mi and mj based on theaddress of mi. Then, mi’s message can be forwarded alongthe observed path along the nodes. The observed path canalways be used by mi for any subsequent messages to Bj aslong as it is not broken. The neighboring BSes of an overloadedBS may also be overloaded. As the mobile nodesnear an overloaded BS know that the BS is overloaded,when they receive a query message to find a path to anunderloaded BS, they do not forward the message towardstheir overloaded BSes.Node mi may receive responses from a few nodes nearBSes. It can choose bðb _ 1Þ neighboring BSes of the destinationto forward the segment. The redundant transmissionsenhance the data transmission reliability while also increasethe routing overhead. Thus, the value of b should be carefullydetermined based on the available resources for routingand the reliability demand. If b is set to a large value, therouting reliability is high at the cost of high overhead. If b isset to a small value, the routing reliability is low while theoverhead is reduced. After the neighboring BSes receive thesegments, they further forward the segments to the destinationBS, which forwards the segments to the destinationnode. In this way, the heavy traffic from mobile nodes to aBS can be distributed among neighboring BSes quickly.Next, we discuss how to handle the case when the destinationBS is congested. If a BS has not received confirmationfrom the destination BS during a certain time periodafter it sends out a segment, it assumes that the destinationBS is overloaded. Then, it sends the segment to b ðb _ 1Þ lightly loaded neighboring BSes of the destination BS fromits routing table. If an attempted neighboring BS does notrespond during a certain time period, it is also consideredas overloaded. Then, the BS keeps trying other neighboringBSes until finding lightly loaded BSes. Redundant neighboringBSes are selected in order to increase routing reliability.The constant b should be set to an appropriatevalue considering factors such as the network size and theamount of traffic in order to achieve an optimal trade-offbetween overhead and reliability.After receiving the message, each lightly loaded neighboringBS of the destination BS finds a multi-hop path to thedestination mobile node. It broadcasts a path query message,which includes the IDs of the destination BS and thedestination node, to the mobile nodes in its region. The pathquerying process is similar to the previous path queryingfor a lightly loaded BS. The nodes further forward the pathquery to their neighbors until the query reaches the destinationnode. Here, we do not piggyback the query to beaconmessages because this querying is for a specific mobile noderather than any mobile node near a lightly loaded BS.Including the mobile node’s ID into beacon messages generatesvery high overhead.In order to reduce the broadcasting overhead, a mobilenode residing in the region of a BS not close to the destinationBS drops the query. The nodes can determine theirapproximate relative positions to BSes by sensing the signalstrengths from different BSes. Each node adds the strengthof its received signal into its beacon message that is periodicallyexchanged between neighbor nodes so that the nodescan identify their relative positions to each other. Only thosemobile nodes that stay farther than the query forwarderfrom the forwarder’s BS forward the queries in the directionof the destination BS. In this way, the query can be forwardedto the destination BS faster. After the multi-hoppath is discovered, the neighboring BS sends the segment tothe destination node along the path. Since the destinationnode is in the neighboring BS’s region, the overhead to identifya path to the destination node is small. Note that ourmethods for congestion control in base stations involvequery broadcasting. However, it is used only when somebase stations are overloaded rather than in the normal DTRrouting algorithm in order to avoid load congestion in BSes.Fig. 4 shows an example of the congestion control onBSes when b ¼ 2. As shown in figure, BS B1 is congested.Then, the relay nodes of the source node’s message broadcastlocally by beacon piggybacking to find multi-hop pathswhich lead to B3 and B4. The relay nodes then send segmentsalong the paths. In this way, the traffic originally targetingoverloaded B1 can be spread out to the neighboringBSes B3 and B4. B3 and B4 further forward the segments tothe destination BS B6 if B6 is not congested. If B6 is also congested,B3 and B4 send the segments to the neighboringBSes of B6. Specifically, B4 sends the segment to B3. B3 doesnot forward the segment to another BS since it already isclose to B6. B3 then finds a multi-hop path to the destinationnode and uses ad-hoc transmission to forward the segmentsto the destination node. Similarly, when B2 wants to send asegment to the destination node, it also uses a multi-hoppath for the segment transmission.4 PERFORMANCE ANALYSIS OF THE DTRPROTOCOLIn this section, we analyze the effectiveness of the DTR protocolat enhancing the capacity and scalability of hybridwireless networks. In our analysis, we use the same scenarioin [17] for hybrid wireless networks, and use the same scenarioin [33] for the ad-hoc network component. We presentthe scenarios and some concepts below. We consider a largenumber of mobile nodes uniformly and randomly deployedover a 2D field. The moving directions of the nodes areindependent and identically distributed (i.i.d.). The distributionof mobile nodes can be modeled as a homogeneousPoisson process with node density s [34]. That is, given anarea of size S in the field, the number of nodes in the area,denoted by nðSÞ, follows the Poisson distribution with theparameter sS,Pr nðSÞ ¼ k ð Þ¼ ðsSÞke_sSk!; k ¼ 0; 1; 2; . . . (1)Fig. 4. Congestion control on BSes.SHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1981Besides mobile nodes, there are M BSes regularly deployedin the field. The BSes divide the area into a hexagon tessellation,in which each hexagon has side length h. The BSes areassumed to be connected together by a wired network. Weassume that the link bandwidths in the wired network arelarge enough so that there are no bandwidth constraintsbetween BSes. In single-path transmission, a message issequentially transmitted in one routing path. In multi-pathtransmission, a message is divided into a number of segmentsthat are forwarded along multiple paths in a distributedmanner. We assume each segment has the same lengthl. Table 1 lists the notations used in our analysis.We assume that the transmission range of all mobilenodes and all BSes is R (R > h). In this paper, we use protocolmodel [17], [33] to describe the interference among nodes;that is, a transmission from a node (here “node” can beeither mobile node or BS) vi to another node vj is successfulif the following two conditions are satisfied: 1) vj is withinthe transmission range of vi, i.e.,jvi _ vjj _ R; (2)where jvi _ vjj represents the euclidean distance between viand vj in the plane. 2) For any other node vk that is simultaneouslytransmitting over the same channel,jvk _ vjj _ ð1 þ DÞjvi _ vjj: (3)Formula (3) guarantees a guard zone around the receivingnode to prevent a neighboring node from transmitting onthe same channel at the same time. The radius of the guardzone is ð1 þ DÞ times the distance between the sender andthe receiver. The parameter D defines the size of the guardzone and we require that D > 0.We first adopt a concept called aggregate throughput capacityintroduced in [17], [33] to measure the throughput of thenetwork.Definition (Aggregate Throughput Capacity of HybridNetworks). The aggregate throughput capacity of a hybridwireless network is of order Qðfðs;MÞÞ if there are deterministicconstants a > 0, and a0 < þ1 such thatlimM!1PrðPðs;MÞ ¼ afðs;MÞ is feasibleÞ ¼ 1 (4)lim infM!1PrðPðs;MÞ ¼ a0fðs;MÞ is feasibleÞ < 1: (5)Since the working frequency of infrastructure networks isaround 700 MHz while that of ad-hoc networks is 2.4 GHz,the communications in infrastructure mode (betweenmobile nodes and BSes through cellular interface) wouldnot generate interference to ad-hoc mode. We divide thechannel for infrastructure mode transmissions into uplinkand downlink parts, according to the transmission directionrelative to the BSes. Accordingly, in the DTR protocol, thetraffic of each S-D pair is composed of at most two intra-celltraffics, one uplink traffic and one download traffic. Sinceuplink traffic and downlink traffic use different sub-channels,there is also no interference between these two types oftraffics. For each node vi, we denote the bandwidth assignedto intra-cell, uplink, and downlink sub-channels by Winti ,Wupi and Wdowni , respectively. We let Wupi ¼ Wdowni becausethere are the same amount of uplink traffic and downlinktraffic. The transmission rates should sum to Wi, i.e.,Winti þWupi þWdowni ¼ Wi. Though no interference existsbetween intra-cell, uplink, and downlink traffics, interferenceexists between the same type of traffic in a cell andbetween different cells. Fortunately, there is an efficient spatialtransmission schedule that can prevent such interferences[17]. First, to avoid the interference in a cell, any twonodes within the cell are not allowed to transmit with thesame traffic mode at the same time. Second, to avoid theinterference between different cells, the cells are spatiallydivided into a number of groups and transmissions in thecells of the same group do not interfere with each other. Ifthe groups are scheduled to transmit in a round robin fashion,each cell will be able to transmit once every fixedamount of time without interfering with each other.Below, we show how many groups we need to divide thecells to prevent interference. We adopt the notion of interferingneighbors introduced in [17], and give the number of cellsthat can be affected by a transmission in one cell. Two cellsare defined to be interfering neighbors if there is a point inone cell which is within a distance (2 þ D)R of a point in theother cell. Accordingly, if two cells are not interfering neighbors,transmissions in one cell do not interfere with transmissionsin the other cell. [17] has proved that (1) each cellhas no more than c1 interfering neighbors (Lemma 1 in[17]), where c1 is a constantc1 ¼433l þ 2R þ DR3l_ _2; (6)and (2) all cells should be divided into c1 þ 1 groups and thewhole channel should be divided into c1 þ 1 subchannels,where each subchannel is allocated to the cells in one group.Thus, the number of group we need to divide the cells toprevent interference is c1 þ 1.Before calculating the aggregate throughput capacity ofDTR, we first introduce Lemma 4.1.Lemma 4.1. The number of cells that have mobile nodes is QðMÞ.Proof. Denote the number of cells having mobile nodesby M1. To prove M1 ¼ QðMÞ, we need to prove thatthere exists deterministic constants a > 0 and a0 < þ1such thatlimM!1PrðM1 ¼ aMÞ ¼ 1; (7)lim infM!1PrðM1 ¼ a0MÞ < 1: (8)TABLE 1Parameter Tables Node density M Number of BSesl Segment’s length sh Area size of a cellnðSÞ Number of nodesin area SR Transmission rangeWi Bandwidth of a node vi mi Mobile node iPðs;MÞ Throughput nðs;MÞ Number of nodes1982 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015For Formula (8), let a0 ¼ 2. Because the number of cellshaving mobile nodes is upper bounded by M, thenlim infM!1PrðM1 ¼ 2M is feasibleÞ ¼ 0: (9)Now, we prove that Formula (7) can also be satisfiedfor some constant a. Because the number of nodes in acell follows a Poisson distribution and the size of eachcell (hexagon) is sh ¼ 3ffiffiffip3h2, then we can derive theprobability that no mobile node is in a cell equalsPr nðShÞ ¼ 0 ð Þ¼s0e_sh0! ¼ e_sh : (10)Consider an arbitrary cell k, let X1;X2, . . ., Xk, . . ., XM bei.i.d. random variables, where Xk represents whether cellk has mobile nodes. Then, Xk is defined as follows:Xk ¼1 cell k has mobile nodes0 cell k does not have mobile nodes_(11)and EðXkÞ ¼ e_sh . For simplicity, let c2 ¼ 1 _ e_sh . Then,M1 ¼PMk¼1 Xk. By theStrong Law of Large Number(SLLN) [34],Pr limM!1PMk¼1 XkM ¼ c2!¼ 1; (12)which implies that limM!1 PrðM1 ¼ c2MÞ ¼ 1, whichindicates that when a ¼ c2, Formula (7) can also besatisfied. tuLemma 4.2. Let nðs;MÞ denote the number of mobile nodes inthe whole network. Then,limM!1Prðnðs;MÞ ¼ shMÞ ¼ 1: (13)Proof. Let Z1, Z2, . . . , ZM be i.i.d. random variables representingthe number of nodes in cell 1, 2, . . . , M, respectively.Then, nðs;MÞ ¼PMk¼1 Zk. Because each Zk followsa Poisson distribution with parameter sh, EðZkÞ ¼ sh,81 _ k _ M. According to SLLN,Pr limM!1PMk¼1 ZkM ¼ sh!¼ 1; (14)which implies that limM!1 PrðPMk¼1 Zk ¼ shMÞ ¼ 1, andhence limM!1 Prðnðs;MÞ ¼ shMÞ ¼ 1. tuTheorem 4.1. For a hybrid network of M BSes and s mobilenode density, where each node has the intra-cell, uplink anddownlink sub-channel bandwidth satisfyingWdowni ¼ Wupi ¼ Wup ¼ W=4; Winti ¼ Wint ¼ W=2 (15)the aggregate throughput capacity of DTR isPðs;MÞ ¼ QðMWÞ: (16)Proof. To prove Pðs;MÞ ¼ QðMWÞ, we need to prove thatthere exists deterministic constants a > 0 and a0 < 1such thatlimM!1PrfPðs;MÞ ¼ aMW is feasibleg ¼ 1 (17)lim infM!1PrfPðs;MÞ ¼ a0MW is feasibleg < 1: (18)Recall that any two nodes within a cell cannot transmitsimultaneously in the same traffic mode, thethroughput P is upper bounded by MW=4, which canbe achieved only if each cell has one node to send themessage. Hence, Formula (18) can be satisfied by settinga0 to 1/2.Then, we will show how Formula (17) can be satisfied.Since the same message has to go through anuplink and a downlink and it is counted only once inthe throughput capacity, calculating the throughput ofthe whole network is equivalent to calculating thethroughput of uplink traffic Pup or the throughput ofdownlink traffic Pdown. Notice calculating intra-celltraffic throughput is not accurate because a messagemay transmit twice with intra-cell mode. In this proof,we calculate Pup.First, we consider the throughput of the uplink trafficof an arbitrary cell k, denoted by Pkup. Since the scheduleallocates 1=ðc1 þ 1Þ time slots to this cell, thenPkup ¼Wupc1 þ 1: (19)Then, we consider the throughput of the whole network.Let Pup ¼PMi¼1 PiupXi represent the throughput of uplinktraffic, then we havelimM!1Pr Pup ¼c2MW3ðc1 þ 1Þ_ _¼ limM!1PrXMi¼1PiupXi ¼c2MWupc1 þ 1!¼ limM!1PrXMi¼1Xi ¼ c2M!¼ 1 ðBy Lemma 4:1Þ:Accordingly, Formula (17) can be satisfied when a isset to c23ðc1þ1Þ. tuCorollary 4.1. With the restriction in Theorem 4.1, DTR canachieve QðWÞ throughput per S-D pair.Proof. Denote the throughput of per S-D pair by P, whichequalsP ¼Pðs;MÞn: (20)Obviously, P is upper bounded by W4 because each nodehas at most W4 for uplink traffic (or downlink traffic),which equals its S-D pair throughput. By Lemma 4.2 andTheorem 4.1, we can derive thatSHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1983limM!1Pr P ¼c2W3ðc1 þ 1Þsh_ _¼ limM!1PrPðs;MÞnðs;MÞ ¼c2W3ðc1 þ 1Þsh_ __ limM!1Pr Pðs;MÞ ¼c2WM3ðc1 þ 1Þ_ _Prðnðs;MÞ ¼ shMÞ¼ limM!1Pr Pðs;MÞ ¼c2WM3ðc1 þ 1Þ_ _¼ 1which implies that limM!1 Pr P ¼ c2W3ðc1þ1Þsh_ _¼ 1. tuCorollary 4.1 shows that DTR produces a constantthroughput for each pair of nodes regardless of the numberof nodes in each cell due to its spacial reuse of thesystem. Theorem 4.1 and Corollary 4.1 show that theaggregate throughput capacity and the throughput perS-D pair of DTR are QðMWÞ and QðWÞ, respectively.The work in [17] proves that DHybrid achieves QðMWÞinfrastructure aggregate throughput, and the work in[33] proves that the pure ad-hoc transmission achievesQð W ffiffiffiffiffiffiffiffiffiffin_logn p Þ throughput per S-D pair. The results demonstratethat the throughput rates of DTR and DHybrid arehigher than that of the pure ad-hoc transmission. This isbecause the pure ad-hoc transmission is not efficient in alarge scale network [35]. A large network size reducesthe path utilization efficiency and increases node interference.Facilitated by the infrastructure network, DTRand DHybrid avoid long distance transmissions, leadingto a higher transmission throughput.Proposition 4.1. Suppose a mobile node needs to allocate totallyU segments with the same length to L neighboring mobilenodes m1, . . ., mL, which has uplink bandwidth Wup1 , . . . ,WupL , respectively. Let Ui denote the number of segments to beallocate to mi (i ¼ 1; 2; . . . ; L). To minimize the averagelatency of these segments, the optimal allocation should satisfyU1Wup1 ¼ _ _ _ ¼ ULWupL. The minimized average latency equalsUl2PLi¼1Wupi.Proof. Recall that each segment has length l. Then, foreach mobile node mi it requires lWupitime to transmit asegment. Therefore, the jth segment that mi needs totransmit has to wait ðj_1ÞlWup1slots. Hence, the totallatency of the segments that mi needs to transmit toits BS equalsXUij¼1ðj _ 1ÞlWupi ¼ ð0 þ 1þ_ _ _þðUi _ 1ÞÞlWupi _U2i l2Wupi: (21)Hence, the average latency of transmitting all the messagesshould bePKi¼1U2i l2Wupi=U. According to Cauchy-Schwarz inequality [34], the average latency is lowerbounded1UXLi¼1U2i l2Wupi ¼l2UPLi¼1WupiXLi¼1U2iWupiXLi¼1Wupi_l2UPLi¼1WupiXLi¼1ffiffiffiffiffiffiffiffiffiU2iWupis ffiffiffiffiffiffiffiffiffiWupiq !2¼Ul2PLi¼1Wupi: (22)WhenffiffiffiffiffiffiffiU21Wup1rffiffiffiffiffiffiffiWup1p ¼ _ _ _ ¼ffiffiffiffiffiffiffiU2LWupLrffiffiffiffiffiffiffiWupLp , or equivalently, U1Wup1 ¼ _ _ _¼ ULWupL, the average segment latencyPLi¼1U2i l2Wupi=U canachieve the minimum value Ul2PLi¼1Wupi. tuProposition 4.1 indicates that forwarding segments to thenearby nodes with the highest capacity can minimize theaverage latency of messages in the cell. It also balancesthe transmission load of the mobile nodes within a cell.Proposition 4.2 A source node in DTR can find relay nodes formessage forwarding with probabilityP1k¼1k_1kckre_crk! , wherecr ¼ pR2.Proof. Let m denote the number of nodes within mi’s transmissionarea and define the indicator variable Qi byQi ¼1 mi is the highest capacity node0 mi is not the highest capacity node_(23)then,Prfmi can find relays for message forwardingg¼X1k¼0Pr Qi ¼ 0jm ¼ k ð ÞPr m ¼ k ð Þ¼X1k¼1k _ 1kckre_crk!:tuProposition 4.2 indicates that in a high-density network,a source node in DTR can find relay nodes for messageforwarding with a high probability. For example, assumethe average number of neighbor nodes of a source node is10. With the daily increasing number of mobile devices,such an assumption is realistic. Then, the probability ofnot being able to find any node in the range of a node is1 _P1k¼1k_1k10ke_10k! _ 0:12, which is very small. Therefore,in a high-density network, a source node can find neighborsfor message forwarding with a high probability.We use DHybrid to denote the group of routing protocolsin hybrid wireless networks that directly combine the adhoctransmission mode and the infrastructure transmissionmode [1], [5], [6], [12], [13], [14], [15], [16], [17], [18].Proposition 4.3. In a hybrid wireless network, the DHybrid routingprotocol leads to load imbalance among the mobile nodes ina cell.Proof. Fig. 5a shows a cell with a BS and a randomly pickedmobile node mi in the range of the BS. The shaded regionrepresents all possible positions of the source nodes that1984 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015choose mi as the relay node in DHybrid. The total trafficpassing through node mi is the sum of the traffic generatedby the nodes in the shaded region. The area ofshaded region isS ¼ sh _ pD2 ð0 < D < hÞ; (24)where D is the distance between the BS and relay nodemi and sh is the area size of a cell. Therefore, the expectedvalue of traffic passing through node mi isW _ s _ ðsh _ pD2Þ ð0 < D < hÞ; (25)where W is the data transmission rate of a sourcenode, and s is the density of the nodes in a region.Equation (25) shows that the traffic passing throughnode mi decreases as D increases. That is, the nodescloser to the BS have a higher load than the nodes stayingat the brim of the cell. tuProposition 4.4. In a hybrid wireless network, DTR achievesmore balanced load distribution among the mobile nodes ineach cell.Proof. The shaded region in Fig. 5b represents all possiblepositions of the source and relay nodes that choose nodemi as relay node. Suppose m neighbor nodes are chosenas relay nodes, then the expected traffic passing throughnode mi is Wm _ s _ pR2 which shows that the traffic goingthrough node mi is independent of its location relative toits BS. Since every node in the cell has an equal probabilityof generating traffic, the traffic load is balancedamong the nodes in the cell. tu5 PERFORMANCE EVALUATIONThis section demonstrates the properties of DTR throughsimulations on NS-2 [36] in comparison to DHybrid [17],Two-hop [19] and AODV [8]. In DHybrid, a node first usesbroadcasting to observe a multi-hop path to its own BSand then forwards a message in the ad-hoc transmissionmode along the path. During the routing process, if thetransmission rate (i.e., bandwidth) of the next hop to theBS is lower than a threshold, rather than forwardingthe message to the neighbor, the node forwards the messagedirectly to its BS. The source node will be notified ifan established path is broken during data transmission. Ifa source sends a message to the same destination nexttime, it uses the previously established path if it is not broken.In the Two-hop protocol, a source node selects thebetter transmission mode between direct transmission andrelay transmission. If the source node can find a neighborthat has higher bandwidth to the BS than itself, it transmitsthe message to the neighbor. Otherwise, it directly transmitsthe message to the BS.Unless otherwise specified, the simulated network consistsof 50 mobile nodes and four BSes. In the ad-hoc componentof the hybrid wireless network, mobile nodes arerandomly deployed around the BSes in a field of1;000 _ 1;000 square meters. We used the Distributed CoordinationFunction (DCF) of the IEEE 802:11 as the MAC layerprotocol. The transmission range of the cellular interfacewas set to 250 meters, and the raw physical link bandwidthwas set to 2 Mbits/s. The transmission power of the ad-hocinterface was set to the minimum value required to keep thenetwork connected for most times, even when nodes are inmotion in the network. Then, the influence of the transmissionrange on different methods’ performance is controlled.Specifically, we set the transmission range through the adhocinterface to 1.5 times of the average distance betweenneighboring nodes, which can be obtained by measuring thesimulated network. We used the two-ray propagation modelfor the physical layer model. Constant bit rate (CBR) wasselected as the traffic mode in the experiment with a rate of640 kbps. In the experiment, we randomly chose four sourcenodes to continuously send messages to randomly chosendestination nodes. The number of channels for each BS wasset to 10. We set the number of redundant routing paths b inSections 3.4 to 1. We assumed that there was no capacitydegradation during transmission between BSes. Thisassumption is realistic considering the advanced technologiesand hardware presently used in wired infrastructurenetworks. There was no message retransmission for failedtransmissions in the experiments.We employed the random way-point mobility model [37]to generate the moving direction, speed, and pause durationof each node. In this model, each node moves to a randomposition with a speed randomly chosen from ð1 _ 20Þ m/s.The pause time of each node was set to 0. We set the numberof segments of a message to the connection degree of thesource node. The simulation warmup time was set to 100 sand the simulation time was set to 1,000 s. We conductedthe experiments five times and used the average value asthe final experimental result. To make the methods comparable,we did not use the congestion control algorithm inDTR unless otherwise indicated.5.1 ScalabilityFig. 6 shows the average throughput measured in kbps perS-D pair of different routing protocols versus the number ofmobile nodes in the system. The figure shows the throughputof DTR remains almost the same with different networksizes. This result conforms to Corollary 4.1. DTR uses distributedmulti-path routing to fully take advantage of thespatial reuse and avoid transmission congestion in a singlepath. Unlike the multi-hop routing in mobile ad-hoc networks,DTR does not need path query and maintenance.Also, it limits the path length to three to avoid problems inlong-path transmission. The throughput of DHybrid andAODV decreases as the number of nodes in the networkincreases. This is mainly because when the network sizeincreases, more beacon messages are generated in theFig. 5. The traffic load in DHybrid and DTR.SHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1985network. Also, the long transmission path also leads to hightransmission interference. Then, nodes in these methodssuffer from intense interference, leading to more transmissionfailure and degraded overall throughput. Also, themobile node increase in the system leads to high networkdynamism, resulting in frequent route re-establishments.The short routing paths in Two-hop reduce congestionand signal interference, thus enabling better spatial reuse asin DTR. Meanwhile, Two-hop enables nodes to adaptivelyswitch between direct transmission and relay transmission.Hence, part of the transmission load is transferred to relaynodes, which carry the messages until meeting the BSes. Asa result, gateway nodes connecting mobile nodes and BSesare not easily overloaded. Therefore, the throughput ofTwo-hop is higher than DHybrid. However, since the numberof message routing hops is confined to one, Two-hopmay not find the node with the best transmission rate to theBSes because of the short transmission range of the ad-hocinterface. Therefore, the throughput of Two-hop is lowerthan DTR, especially in a network with high node density.The reason that AODV has the lowest throughput per S-Dpair is its long transmission paths.Fig. 7 shows the throughput per S-D pair versus thenumber of BSes in different routing protocols. The numberof BSes was varied from 3 to 6. The BSes are uniformly distributedin the network. We can see from the figure that asthe number of BSes increases, the throughputs of DTR,Two-hop, and DHybrid increase while the throughput ofAODV stays nearly constant. In DTR, Two-hop, and DHybrid,as the number of BSes increases, the total number ofnodes close to the BSes increases. Then, more nodes havehigh transmission rates to the BSes, leading to a throughputincrease. In AODV, since the traffic between S-D pairs doesnot travel though BSes, the throughput between an S-D pairis not affected by the increased number of BSes in the network.The figure also shows that the throughput of DTR isconstantly larger than Two-hop and the throughput ofTwo-hop is constantly larger than DHybrid. AODV constantlyhas the lowest transmission delay due to the samereasons as in Fig. 6.5.2 Transmission DelayFig. 8 shows the average transmission delay of S-D pairsfor successfully delivered messages in different routingprotocols versus network size. The network size was variedfrom 20 to 100 with 20 increase in each step. Transmissiondelay is the amount of time it takes for a message tobe transmitted from its source node to its destinationnode. From the figure, we see that DTR generates thesmallest delay. In DTR, each source node first dividesits messages into smaller segments and then forwardsthem to the nearby nodes with the highest capacity, whichleads to more balanced transmission load distributionamong nodes than the previous methods. According toProposition 4.1, average latency can be minimized whenthe transmission loads of all the nodes are balanced.Hence, DTR has smaller latency than the previous methods.The delay of DHybrid is five to six times larger thanDTR. DHybrid uses a single transmission path, while DTRuses multiple paths. Recall that we set the number of segmentsof a message to the connection degree of the sourcenode in DTR. Thus, the ratio of delay time of DHybrid tothat of DTR equals the average connection degree. As thenumber of nodes in the system increases, the connectiondegree of each node increases, and the increase rate of theratio grows. This is caused by two reasons. First, a highernode density leads to longer path lengths in DHybrid,resulting in a longer delay because of a higher likelihoodof link breaks. Second, a higher node density enables anode to quickly find relay nodes to forward messages inDTR, as indicated in Proposition 4.2.DTR also produces a shorter transmission delay thanTwo-hop for two reasons. First, the multi-path parallel routingof DTR saves much transmission time as shown inProposition 4.1. Second, the distributed routing of DTR enablessome messages to be forwarded to the destination BS’sneighboring cells with high transmission rates rather thanwaiting in the current hot cell for a transmission channel.We can also observe that Two-hop produces lower delaythan DHybrid. This is because the delay of DHybirdincludes the time for establishing a path and for data transmission.Also, the multi-hop transmission component ofDHybrid results in a higher delay due to the queuing delayin each hop. Because of the long distance transmissionswithout support from an infrastructure network, AODVgenerates the longest delay.Fig. 6. Throughput vs. network size (simulation).Fig. 7. Throughput vs. number of BSes.Fig. 8. Delay vs. network size.1986 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015Fig. 9 plots the average communication delay per S-Dpair for successfully delivered messages versus the numberof BSes in different routing protocols. The figure shows thatthe increasing number of BSes in the system leads to a communicationdelay decrease between nodes in DTR, Twohop,and DHybrid, but does not affect the communicationdelay in AODV. In DTR, Two-hop, and DHybrid, as thenumber of BSes increases, more nodes can stay close to theBSes, leading to fewer communication hops and bettertransmission links between nodes and BSes. Thus, the transmissiondelay between the nodes is reduced. Since thecommunication between S-D pairs in AODV does notrely on BSes, AODV maintains constant communicationdelay. The figure also shows that the communication delaybetween S-D pairs follows DTR<Two-hop<DHybrid<AODV for the same reason as in Fig. 8.5.3 Communication OverheadWe use the generation rate of control messages in the networkand MAC layers in kbps to represent the communicationoverhead of the routing protocols. Fig. 10 illustrates thecommunication overhead of DTR, Two-hop, DHybrid, andAODV versus network size. We can see that the communicationoverheads of DTR and Two-hop are very close. Thisis because both DTR and Two-hop are transmission protocolsof short distance and small hops. DTR has slightlyhigher communication overhead than Two-hop becauseDTR utilizes three hop transmission, which has one morehop than two hop transmission. However, the marginaloverhead increase leads to a much higher transmissionthroughput as shown in Fig. 6. DHybrid generates muchhigher overhead than DTR and Two-hop because of thehigh overhead of routing path querying. The pure AODVrouting protocol results in much more overhead than theothers. This is because without an infrastructure network,the messages in AODV travel a long way from the sourcenode to the destination node through much longer paths.5.4 Effect of MobilityIn order to see how the node mobility influences the performanceof the routing protocols, we evaluated the throughputof these four transmission protocols with different nodemobilities. Fig. 11 plots the throughput of DTR, DHybrid,Two-hop, and AODV versus node moving speed. From thefigure, we can see that the increasing mobility of the nodesdoes not adversely affect the performance of DTR and Twohop.It is intriguing to find that high mobility can even helpDTR to increase its throughput and that Two-hop generatesconstant throughput regardless of the mobility. This isbecause the DTR and Two-hop transmission modes do notneed to query and rely on multi-hop paths; thus, they arenot affected by the network partition and topology changes.Moreover, since DTR transmits segments of a message in adistributed manner, as the mobility increases, a mobilenode can meet more nodes in a shorter time period. Therefore,DTR enables the segments to be quickly sent to highcapacitynodes. As node mobility increases, the throughputof DHybrid decreases. In DHybrid, the messages are routedin a multi-hop fashion. When the links between nodes arebroken because of node mobility, the messages are dropped.Therefore, when nodes have smaller mobility, the linksbetween the mobile nodes last longer and more messagescan be transmitted. Hence, the throughput of DHybrid isadversely affected by node mobility. However, since DHybridcan adaptively adjust the routing between the ad-hoctransmission and cellular transmission, the throughput ofDHybrid is much higher than AODV’s. With no infrastructurenetwork, AODV produces much lower throughputthan the others. Its throughput also drops as node mobilityincreases for the same reasons as DHybrid.5.5 Effect of WorkloadWe measured the total throughput of BSes on the messagesreceived by BSes. Fig. 12 shows the total throughput of theBSes versus the number of source nodes. We can see thatDTR and Two-hop have much higher throughput increaserates than DHybrid. This is because in DTR and Two-hop,the number of transmission hops from a source node to a BSis small. Meanwhile, each node can adaptively switchbetween relay transmission and direct transmission basedon the transmission rate of its neighbors. Hence, part of asource node’s transmission load is transferred to a few relaynodes, which carry the messages until meeting the BSes.Therefore, the gateway mobile nodes are less likely to becongested. However, nodes in DHybrid cannot adaptivelyadjust the next forwarding hop because it is predeterminedin the routing path. Messages are always forwarded to theFig. 9. Delay vs. number of BSes.Fig. 10. Overhead vs. network size.Fig. 11. Throughput vs. mobility.SHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1987mobile gateway nodes that are closer to the BSes or thathave higher transmission rates. Therefore, these mobilegateway nodes can easily become congested as the workloadof the system increases, leading to many messagedrops. Therefore, when the number of the source nodes islarger than 4, the throughput of DHybrid remains nearlyconstant. This is also the reason that the throughput of DHybridis constantly lower than those of DTR and Two-hop.Additionally, the figure shows that the overall throughputof Two-hop is lower than that of DTR. This is because mostof the traffic in Two-hop is confined to a single cell. When aBS in a cell is congested, the traffic cannot be transferred toother cells. In contrast, DTR’s three-hop distributed forwardingmechanism enables it to distribute the trafficamong the BSes in a balance. Therefore, the BSes in DTRwill not become congested easily. In addition, as the forwardingmechanism gives nodes more flexibility in choosingrelay nodes with higher transmission rates for messageforwarding to the BSes, the overall BS throughput in DTR islarger than in Two-hop.5.6 Effect of the Number of Routing HopsWe conducted experiments to show the optimal number ofrouting hops for the routing in hybrid wireless networks.We tested the throughput per S-D pair for x-hop DTR,where x was varied from 1 to 4. In the one-hop routing, anode directly transmits a message to the BS without messagedivision. In the other routing protocols, the ðx _ 1Þthhop chooses the best transmission mode between directtransmission and relay transmission. Also, in the four-hoprouting, the second relay node randomly chooses the thirdrelay node.Fig. 13 shows the average throughput per S-D pair versusnetwork size in DTR. As the figure shows, as the networksize increases, the node throughput keeps constant regardlessof the number of forwarding hops in a routing. The reasonis the same as in Fig. 6. We can also see from the figurethat the throughput of the four protocols follows 3-hop>4-hop>2-hop>1-hop. In the one-hop routing, each node onlytransmits segments directly to a BS regardless of its currenttransmission rate. In the two-hop routing, if the transmissionrate of a node’s neighbor is higher than that of thenode, it asks its neighbor node to forward the segment to aBS. Therefore, the two-hop routing has higher throughputthan the one-hop routing. The three-hop routing can greatlyincrease the number of node options for segment routingsince the number of nodes that the source node can encounterincreases from d to d2, where d is the average nodedegree. Thus, a node with a greater transmission rate can bechosen as the forwarding node. Meanwhile, the three-hoprouting can greatly facilitate inter-cell communicationbecause a node has a higher probability of reaching aneighboring BS within a three-hop path length than withina two-hop path length. Therefore, the throughput of thethree-hop routing is much higher than that of the two-hoprouting. The figure also shows that the four-hop routingproduces lower throughput than the three-hop routing. Thereason is that three hops are enough to find a hop with hightransmission rate and achieve inter-cell communicationbecause of widespread BSes. The four-hop routing increasesthe forwarding delay due to the greater number of nodes ina route; thus, it cannot increase the uploading transmissionrate of messages.5.7 Load Distribution within a CellIn this experiment, we tested the load distribution of mobilenodes in a randomly chosen cell in the hybrid wireless networkthat employs each of the DTR, DHybrid, and Twohopprotocols. We normalized the distance from a mobilenode to its base station according to the function DRb, whereD is the actual distance and Rb is the radius of its cell. Wedivided the space of the cell into several concentric circlesand measured the loads of the nodes on each circle to showthe load distribution.Fig. 14 shows the average load of a node correspondingto the normalized distance from itself to the BS in the chosencell. The figure shows that most of the traffic load of DHybridis located at nodes near the BS. The nodes far from theBS have very low load. The results conform to Proposition4.3. In DHybrid, if a source node wants to access the Internetbackbone or engage in inter-cell communication, it transmitsthe messages to the BSes in a multi-hop fashion. Therefore,the nodes near the BSes will have the highest load. Onthe other hand, since there is little traffic going through thenodes at the brim of a cell, the load of these nodes is small.As a result, some nodes can easily become hot spots whileFig. 12. Throughput of BSes vs. number of source nodes.Fig. 13. Throughput vs. number of hops.Fig. 14. Load distribution in a cell.1988 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015the resources of other nodes are not fully utilized. This loadimbalance prevents DHybrid from fully utilizing systemresources. The traffic load of DTR is almost evenly distributedin the system, which is in line with Proposition 4.4. InDTR, the traffic from a source node is distributed among anumber of relay neighbors for further data forwarding. Thenodes at the brim of the cell also take responsibility for messageforwarding, since the neighbor nodes of the brimnodes could be located in other cells with good transmissionchannels. In Two-hop, the source node considers directtransmission or one-hop relay transmission based on thechannel condition. Since the node is chosen within one hop,the messages will not gather close to the BS due to the limitedtransmission range. However, because of its sequentialtransmission, Two-hop cannot achieve load balance amongnodes in a cell as well as DTR.5.8 Load Balance between CellsIn this experiment, we tested the effectiveness of the congestioncontrol algorithm in DTR. We also added a congestioncontrol algorithm to DHybrid. In the algorithm, when anode receives beacon messages from its BS indicating that itis overloaded, the node broadcasts a query message to finda path to a nearby uncongested BS. We selected two BSesout of the total four BSes. In the range of each of the twoselected BSes, we randomly selected one mobile node as thesource node to send messages to a randomly selected destinationnode in the network. Once the source node movesout of the range of the selected BS, another mobile node inthe range was selected as the source node. In order to showthe load distribution of the BSes in different protocols, weranked the BSes based on BS throughput. The BS with thehighest throughput has a rank of 1.Fig. 15 shows the throughput of each BS versus the BSrank. We can see from the figure that in Two-hop, thethroughput of the first two BSes is extremely high whilethe throughput of the last two BSes is extremely small.This is because the two hop routing path length in Twohopis not long enough to forward messages from a congestedBS to a lightly loaded BS. Therefore, the traffic cannotbe shifted to the neighboring lightly loaded BSes,leading to an unbalanced load distribution. We can alsosee from the figure that in DTR, the variance of thethroughputs in different BSes is small. The reason is thatthree forwarding hops are enough for a mobile node toreach a neighboring BS and hence to balance the loadbetween the BSes. Meanwhile, the congestion control algorithmin DTR can effectively switch the traffic from ahighly loaded cell to a lightly loaded cell. Because theBSes of ranks 1 and 2 in DTR are not congested, theirthroughput is less than the corresponding BSes in Twohop;also, the throughput of the BSes of ranks 3 and 4 inDTR is much higher than that of the corresponding BSesin Two-hop. DHybrid achieves more balanced load distributionbetween BSes than Two-hop since it employs a congestioncontrol algorithm. In DHybrid, if a previouslyestablished path to a destination is not broken, a node stilluses this path to transmit messages to the same destination.Thus, the nodes cannot dynamically balance loadbetween BSes. Also, when a node finds that its current BSis congested, it takes a long time for it to find a path to anon-congested BS by re-issuing a query message to theneighboring non-congested BS, which greatly reduces thethroughput of the system.Fig. 16 further shows the throughput of the BSes versussimulation time in the three routing protocols. At the beginning,the BSes with ranks 1 and 2 are congested and thosewith ranks 3 and 4 do not have much traffic. Thus, the threefigures show that the BSes with ranks 1 and 2 have highthroughput but those with ranks 3 and 4 have extremelylow throughput at the beginning in all three protocols.Fig. 16a shows the throughput of the BSes in DTR. Asshown in the figure, since DTR can adaptively adjust thetraffic among the BSes using its congestion control algorithm,the throughput of the two highly congested BSes isdistributed to the neighboring BSes. As the traffic is forwardedfrom the BSes of ranks 1 and 2 to the BSes of ranks 3and 4, the throughputs of these BSes are very similar later inthe simulation. This result indicates the effectiveness of thecongestion control algorithm in DTR for load balancebetween cells.Fig. 16b shows the throughput of the BSes in Two-hop. InTwo-hop, since the source nodes cannot effectively movethe traffic between BSes, the BSes with rank 1 and rank 2Fig. 15. Load distribution among BSes.Fig. 16. Base station load vs. simulation time.SHEN ET AL.: A DISTRIBUTED THREE-HOP ROUTING PROTOCOL TO INCREASE THE CAPACITY OF HYBRID WIRELESS NETWORKS 1989constantly have the highest throughput, while the BSes withrank 3 and rank 4 constantly have low throughput. The lowthroughput is produced when the immediate neighbors ofthe source node are in the range of the neighboring BSes ofthe source node’s BS. However, the probability of such casesis very small. Fig. 16c shows the throughput of the BSes inDHybrid. As the nodes in DHybrid cannot effectively balancethe load between the BSes, the throughput of the BSesof rank 1 and rank 2 is much larger than that of the BSes ofrank 3 and rank 4. Comparing Figs. 16b and 16c, we canfind that the throughput in DHybrid is lower than that inTwo-hop. This is because the multi-hop transmission in thead-hoc network in DHybrid greatly reduces the throughput.Meanwhile, the mobile gateway nodes in DHybrid easilybecome congested, leading to more message drops.6 CONCLUSIONSHybrid wireless networks have been receiving increasingattention in recent years. A hybrid wireless network combiningan infrastructure wireless network and a mobile adhocnetwork leverages their advantages to increase thethroughput capacity of the system. However, currenthybrid wireless networks simply combine the routing protocolsin the two types of networks for data transmission,which prevents them from achieving higher system capacity.In this paper, we propose a Distributed Three-hop Routingdata routing protocol that integrates the dual features ofhybrid wireless networks in the data transmission process.In DTR, a source node divides a message stream into segmentsand transmits them to its mobile neighbors, whichfurther forward the segments to their destination throughan infrastructure network. DTR limits the routing pathlength to three, and always arranges for high-capacitynodes to forward data. Unlike most existing routing protocols,DTR produces significantly lower overhead by eliminatingroute discovery and maintenance. In addition,its distinguishing characteristics of short path length, shortdistancetransmission, and balanced load distribution providehigh routing reliability and efficiency. DTR also has acongestion control algorithm to avoid load congestion inBSes in the case of unbalanced traffic distributions in networks.Theoretical analysis and simulation results showthat DTR can dramatically improve the throughput capacityand scalability of hybrid wireless networks due to its highscalability, efficiency, and reliability and low overhead.ACKNOWLEDGMENTSThis research was supported in part by US NSF grants NSF-1404981, IIS-1354123, CNS-1254006, CNS-1249603, andCNS-1025652, and Microsoft Research Faculty Fellowship8300751. The authors would like to thank Mr. Kang Chenfor his help in addressing review comments. An early versionof this work was presented in the Proceedings of ICPP2009 [38]. Haiying Shen is the corresponding author.