Saturday, July 25, 2020

A few words about ConcurrentHashMap, compute() method

There is one ambiguous moment in the Java documentation of the compute() method from the ConcurrentHashMap class:

Some attempted update operations on this map by other threads may be blocked while computation is in progress, so the computation should be short and simple, and must not attempt to update any other mappings of this Map.

It's not clear what is being blocked. Is it the access to the entire map or to the key? Let's find out, by conducting a few experiments with the following code:

package tests.concurrency;

import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;

public class TestConcurrentHashMapCompute implements Runnable {
	private static final Map<Key, String> MAP = new ConcurrentHashMap<>();
	private static final int MAX_RUNS = 100;
	private static final long SLEEP_TIME = 5000;

	public static void main(String[] args) {
		Thread th1 = new Thread(new TestConcurrentHashMapCompute());
		Thread th2 = new Thread(new TestConcurrentHashMapCompute());

		th1.start();
		th2.start();
	}

	@Override
	public void run() {
		final String name = Thread.currentThread().getName();
		for (int i = 0; i < MAX_RUNS; i++) {
			final String value = MAP.compute(new Key(name), (k, v) -> {
				System.out.println(name + ": enters");
				sleep();
				System.out.println(name + ": exits");
				return name;
			});
			System.out.println(name + ": " + value);
		}
	}

	private static void sleep() {
		try {
			Thread.sleep(SLEEP_TIME);
		} catch (Exception ex) {}
	}

	private static class Key {
		private final String value;

		private Key(String value) {
			this.value = value;
		}

		@Override
		public int hashCode() {
			return 1;
			// return Objects.hash(value);
		}

		@Override
		public boolean equals(Object o) {
			if (this == o) {
				return true;
			}
			if (o == null || getClass() != o.getClass()) {
				return false;
			}
			Key other = (Key) o;
			return Objects.equals(value, other.value);
		}
	}
}

It's nothing complicated, two threads calling the compute() method of a shared ConcurrentHashMap instance, using Key class instances as the key for the map. However, pay attention to the hashCode() method inside the Key class:

		@Override
		public int hashCode() {
			return 1;
			// return Objects.hash(value);
		}

Depending on what is being returned (comment/uncomment the relevant lines), you will see different behaviour. And, by the way, return 1 is a valid hashCode() implementation, according to the Java documentation:

- If two objects are equal according to the equals() method, then calling the hashCode method on each of the two objects must produce the same integer result.
- It is not required that if two objects are unequal according to the equals() method, then calling the hashCode method on each of the two objects must produce distinct integer results. However, the programmer should be aware that producing distinct integer results for unequal objects may improve the performance of hash tables.

The result is:
- With return Objects.hash(value), i.e. String's descent hashCode() implementation, the threads run in parallel, as if the block was on the Key.
- With return 1, the threads run sequentially, as if the block was on the entire Map.

In fact, the block is on the bucket indexed by the Key. That's what the hashCode() method is about, to quickly access the relevant bucket. HashMap, for example, uses a singly linked list as the storage for the buckets.

Wednesday, July 3, 2019

401 and/or 403 and a short story of secure RESTful

The problem of what HTTP codes to use inevitably comes when designing RESTful services. There is a bit of subtlety, however, with regards to the 401 and 403 status codes. Here and there you will read advices suggesting to use:

  • 401 for when an access token isn’t provided, or is invalid.
  • 403 for when an access token is valid, but requires more privileges.
and people will follow these advices without considering the security part of the problem.

And the problem is that such an implementation can leak sensitive information. Here is my reply to the above mentioned article. In cryptography this is known as oracle and can lead to pretty serious attacks.

If the attacker sees a 403 status code, after a huge array of failed attempts with 401, this means that whatever the attacker tried passed the authentication phase, but failed the authorisation. Bingo, they just figured out a valid set of credentials (or tokens or ...) and your system said that in "clear text". It is similar to the "Username or password invalid" best practices (and don't read things like this!).

How to avoid the problem? Stick with one status code (e.g. 403) for both cases (failed authentication and failed authorisation), avoid being too user (or client) friendly. You can log the incident and return a unique request ID to the client. If a genuine client reports the problem, that request ID should help finding more details in logs. And of course track/monitor all the authentication/authorisation failures for anomaly detection purposes.

Now the fun part, all of the above as a mathematical proof. Let's assume a system where the maximum number of credentials possible is $N$, has $M$ registered users and $K$ of those users have access to a specific resource that the attacker tries to brute-force, $N>M\geq K$. Let's consider the following propositions/events:

  • $A$ - guess a credential by accessing the given resource.
  • $B$ - The system is designed to return: HTTP 200 for valid credentials with access to the resource, HTTP 403 for valid credentials with no access to the resource, HTTP 401 for invalid credentials. For simplicity, we can say $B=\{200\}\bigcup \{403\}\bigcup \{401\}$.
  • $C$ - The system is designed to return: HTTP 200 for valid credentials with access to the resource, HTTP 403 for valid credentials with no access to the resource or invalid credentials. Or $C=\{200\}\bigcup \{403\}$

In other words:

  • Cardinality of $A$ givnen $B$ is: $K$ possible cases of HTTP 200, plus $M-K$ possible cases of HTTP 403. Grand total is $M$.
  • Cardinality of $A$ givnen $C$ is: $K$ possible cases of HTTP 200, which is also the grand total.

Now let's compute probabilities: $$P(A \mid B)=\frac{M}{N}$$ and $$P(A \mid C)=\frac{K}{N}$$ Obviously (because $M \geq K$) $$P(A \mid B) \geq P(A \mid C)$$ which means, a system designed like $B$ gives greater chances to the attacker to guess credentials. Obviously, it doesn't matter when all the registered users have access to the resource (i.e. $K=M$). End of discussion!

P.S. The way I computed probabilities may look a bit superficial, here is another way using total probabilities or $$P(A\mid B) = \frac{P(A \cap B)}{P(B)}=\sum\limits_{C_n} \frac{P(A \cap B \cap C_n)}{P(B)}=\\ \sum\limits_{C_n} \frac{P(A \cap B \cap C_n)}{P(B\cap C_n)}\cdot \frac{P(B\cap C_n)}{P(B)}=\\ \sum\limits_{C_n} P(A \mid B \cap C_n)\cdot P(C_n\mid B)$$ Then:

  • $$P(A\mid B)=P(A\mid B \cap \{200\})\cdot P(\{200\}\mid B)+\\ P(A\mid B \cap \{403\})\cdot P(\{403\}\mid B)+\\ P(A\mid B \cap \{401\})\cdot P(\{401\}\mid B)=\\ P(A\mid \{200\})\cdot P(\{200\}\mid B)+ P(A\mid \{403\})\cdot P(\{403\}\mid B)+\\ P(A\mid \{401\})\cdot P(\{401\}\mid B)=\\ 1\cdot P(\{200\}\mid B)+ 1\cdot P(\{403\}\mid B)+ 0\cdot P(\{401\}\mid B)=\\ \frac{K}{N}+\frac{M-K}{N}=\frac{M}{N} $$
  • $$P(A\mid C)=P(A\mid C \cap \{200\})\cdot P(\{200\}\mid C)+\\ P(A\mid C \cap \{403\})\cdot P(\{403\}\mid C)=\\ P(A\mid \{200\})\cdot P(\{200\}\mid C)+ P(A\mid \{403\})\cdot P(\{403\}\mid C)=\\ 1\cdot P(\{200\}\mid C)+ 0\cdot P(\{403\}\mid C)=\frac{K}{N}$$

Saturday, September 22, 2018

Generating Subsets

Every once in a while I am being challenged with problems involving optimal subsets from a given set of objects. It happened again a few weeks ago. The problem was of a knapsack style (finding the most profitable subset of items of bounded total size) and ... of course I forgot how to tackle it using dynamic programming. But, many years ago I learned a very simple (to memorise) algorithm which saved the day and I will describe it in this article.

The Algorithm Design Manual, page $453$, defines it as binary counting. The idea behind it is the one to one mapping between the numbers $0 \rightarrow 2^{n-1}$ (total number of numbers is $2^n$) and binary strings of length $n$ (total number of strings is $2^n$) where bit $1$ at the index $i$ means "consider the object at the index $i$ in the original set for the given subset". We also know that the total number of subsets of a set of size $n$ is $2^n$. We have $2^n$ everywhere, awesome!

Let's see it in action by looking at a simple example, a small set $\left\{A, B, C\right\}$ with $n=3$. We are not interested in the empty set, thus we will ignore the $0$-th iteration, leaving us with a total of $2^3-1=7$ subsets:

IterationBinary string Indexes of the objects to considerSubset
1001 1$\left\{A\right\}$
2010 2$\left\{B\right\}$
3011 1 and 2$\left\{A, B\right\}$
4100 3$\left\{C\right\}$
5101 1 and 3$\left\{A, C\right\}$
6110 2 and 3$\left\{B, C\right\}$
7111 1, 2 and 3$\left\{A, B, C\right\}$

I need to stress the fact that it's an exponential (or $O\left(2^n\right)$) complexity algorithm. It is very inefficient for big sets. Here is a code example:

import java.util.LinkedHashSet;
import java.util.Set;

public class AllSubSets {
 public static void main(String args[]) {
  String[] setValues = new String[] { "A", "B", "C", "D", "E",
    "F", "G", "H", "I", "J" };

  long limit = 1 << setValues.length; // this is 2^length

  for (long l = 1; l < limit; l++) {
   Set subSet = new LinkedHashSet<>();
   for (int i = 0; i < setValues.length; i++) {
    if ((l & (1 << i)) > 0) {
     subSet.add(setValues[i]);
    }
   }
   subSet.forEach(System.out::print);
   System.out.println();
  }
 }
}

and the result:


A
B
AB
C
AC
BC
ABC
D
... (trimmed due to the size) ...
CDEFGHIJ
ACDEFGHIJ
BCDEFGHIJ
ABCDEFGHIJ

A careful reader will almost immediately reply that this version is limited to sets with maximum 64 objects (with Java 8, long is limited to 64 bits and can be treated as unsigned). BigInteger, which supports bitwise operations, addresses this limitation. And yes, the extra subset construction loop makes it an $O(n\cdot2^n)$ algorithm, it's still exponential though.

However, as it was mentioned earlier, this is not an efficient algorithm. Anything bigger than 64 objects will produce more than $2^{64}$ subsets and this is a big number. Even $2^{32}$ is pretty decent in size, so don't use this algorithm for big sets.

Monday, August 13, 2018

To Optional or not to Optional

I recently had an interesting conversation with my colleagues about the usage of Optional's in Java. It all revolves around this article.

So, I quickly sketched the following code to highlight the idea:

import java.util.Optional;

public class OptionalEvaluationTest {
 public static void main(String[] args) {
  test1(); // non lazy execution of orElse().
  test2();
  test3();
  test4();
 }

 private static void test1() {
  System.out.println("Test 1");
  Optional<Integer> oi = Optional.ofNullable(3);
  oi.map(OptionalEvaluationTest::intToString)
   .orElse(defaultString());
  System.out.println("----");
 }

 private static void test2() {
  System.out.println("Test 2");
  Optional<Integer> oi = Optional.ofNullable(null);
  oi.map(OptionalEvaluationTest::intToString)
   .orElse(defaultString());
  System.out.println("----");
 }

 private static void test3() {
  System.out.println("Test 3");
  Optional<Integer> oi = Optional.ofNullable(3);
  oi.map(OptionalEvaluationTest::intToString)
   .orElseGet(OptionalEvaluationTest::defaultString);
  System.out.println("----");
 }

 private static void test4() {
  System.out.println("Test 4");
  Optional<Integer> oi = Optional.ofNullable(null);
  oi.map(OptionalEvaluationTest::intToString)
   .orElseGet(OptionalEvaluationTest::defaultString);
  System.out.println("----");
 }

 private static String intToString(int i) {
  System.out.println("converter called!");
  return Integer.toString(i);
 }

 private static String defaultString() {
  System.out.println("default called!");
  return "default";
 }
}

which yields the following result:


Test 1
converter called!
default called!
----
Test 2
default called!
----
Test 3
converter called!
----
Test 4
default called!
----

What we see is that .orElse() is executed always, aka is not lazy executed. This may become a performance issue if used unwisely. Tend to prefer .orElseGet() instead.

Saturday, August 12, 2017

The matter of squares and rectangles

Being in Oxford and not dedicating some time to browse the local book shops is, imho, a crime. In one of my recent visits I found this recreational maths book "Mathematics and Chess" (Dover Publications, 1997) by Miodrag Petković. The very first problem in this book is:

Prove that the total number of rectangles that can be found on a generalized $n \times n$ board is a perfect square of a natural number.

You know, those squares from job interviews and puzzles? Yes, those! So this almost popular matter needs to be sorted out. I will skip the proofs, since almost everything is on Wikipedia these days and the proofs aren't complicated, simple applications of induction. The formula for rectangles is simply beautiful: $$\sum_{k=1}^{n} k^3=\left ( \sum_{k=1}^{n} k \right )^2=\left [ \frac{n(n+1)}{2} \right ]^2$$ Squares and cubes, amazing, isn't it?

While the formula for squares is simply: $$\sum_{k=1}^{n} k^2=\frac{n(n+1)(2n+1)}{6}$$

Saturday, December 31, 2016

To the good people who passed away and those who survived the year

Sometimes I think,
Those people left
Just to avoid seeing
The way this world
Becomes insane
And horribly crazy.

Of course I'm wrong
All people die,
Inevitable matter.
But we survived
And they have left
An obvious reminder.