Idit Keidar

David Yakira, Avi Asayag, Ido Grayevsky et al.

We study the problem of providing blockchain applications with \emph{economically viable randomness} (EVR), namely, randomness that has significant economic consequences. Applications of EVR include blockchain-based lotteries and gambling. An EVR source guarantees (i) secrecy, assuring that the random bits are kept secret until some predefined condition indicates that they are safe to reveal (e.g., the lottery's ticket sale closes), and (ii) robustness, guaranteeing that the random bits are published once the condition holds. We formalize the EVR problem and solve it on top of an Ethereum-like blockchain abstraction, which supports smart contracts and a transferable native coin. Randomness is generated via a distributed open commit-reveal scheme by game-theoretic agents who strive to maximize their coin holdings. Note that in an economic setting, such agents might profit from breaking secrecy or robustness, and may engage in side agreements (via smart contracts) to this end. Our solution creates an incentive structure that counters such attacks. We prove that following the protocol gives rise to a stable state, called Coalition-Proof Nash Equilibrium, from which no coalition comprised of a subset of the players can agree to deviate. In this stable state, robustness and secrecy are satisfied. Finally, we implement our EVR source over Ethereum.

Oded Naor, Idit Keidar

State Machine Replication (SMR) solutions often divide time into rounds, with a designated leader driving decisions in each round. Progress is guaranteed once all correct processes synchronize to the same round, and the leader of that round is correct. Recently suggested Byzantine SMR solutions such as HotStuff, Tendermint, and LibraBFT achieve progress with a linear message complexity and a constant time complexity once such round synchronization occurs. But round synchronization itself incurs an additional cost. By Dolev and Reischuk's lower bound, any deterministic solution must have $Ω(n^2)$ communication complexity. Yet the question of randomized round synchronization with an expected linear message complexity remained open. We present an algorithm that, for the first time, achieves round synchronization with expected linear message complexity and expected constant latency. Existing protocols can use our round synchronization algorithm to solve Byzantine SMR with the same asymptotic performance.

Naama Kraus, David Carmel, Idit Keidar

Similarity search is the task of retrieving data items that are similar to a given query. In this paper, we introduce the time-sensitive notion of similarity search over endless data-streams (SSDS), which takes into account data quality and temporal characteristics in addition to similarity. SSDS is challenging as it needs to process unbounded data, while computation resources are bounded. We propose Stream-LSH, a randomized SSDS algorithm that bounds the index size by retaining items according to their freshness, quality, and dynamic popularity attributes. We analytically show that Stream-LSH increases the probability to find similar items compared to alternative approaches using the same space capacity. We further conduct an empirical study using real world stream datasets, which confirms our theoretical results.

Naama Kraus, David Carmel, Idit Keidar

Today's search engines process billions of online user queries a day over huge collections of data. In order to scale, they distribute query processing among many nodes, where each node holds and searches over a subset of the index called shard. Responses from some nodes occasionally fail to arrive within a reasonable time-interval due to various reasons, such as high server load and network congestion. Search engines typically need to respond in a timely manner, and therefore skip such tail latency responses, which causes degradation in search quality. In this paper, we tackle response misses due to high tail latencies with the goal of maximizing search quality. Search providers today use redundancy in the form of Replication for mitigating response misses, by constructing multiple copies of each shard and searching all replicas. This approach is not ideal, as it wastes resources on searching duplicate data. We propose two strategies to reduce this waste. First, we propose rSmartRed, an optimal shard selection scheme for replicated indexes. Second, when feasible, we propose to replace Replication with Repartition, which constructs independent index instances instead of exact copies. We analytically prove that rSmartRed's selection is optimal for Replication, and that Repartition achieves better search quality compared to Replication. We confirm our results with an empirical study using two real-world datasets, showing that rSmartRed improves recall compared to currently used approaches. We additionally show that Repartition improves over Replication in typical scenarios.

Naama Kraus, David Carmel, Idit Keidar et al.

We present NearBucket-LSH, an effective algorithm for similarity search in large-scale distributed online social networks organized as peer-to-peer overlays. As communication is a dominant consideration in distributed systems, we focus on minimizing the network cost while guaranteeing good search quality. Our algorithm is based on Locality Sensitive Hashing (LSH), which limits the search to collections of objects, called buckets, that have a high probability to be similar to the query. More specifically, NearBucket-LSH employs an LSH extension that searches in near buckets, and improves search quality but also significantly increases the network cost. We decrease the network cost by considering the internals of both LSH and the P2P overlay, and harnessing their properties to our needs. We show that our NearBucket-LSH increases search quality for a given network cost compared to previous art. In many cases, the search quality increases by more than 50%.