I came across 2 older answers (2 different but similar questions on the small cofactor attack) which cover this attack.
Here the attacker replaces the $bG$ Bob sends to Alice with $H$ where $H$ is a point of smaller order.
Here the attacker replaces the $bG$ Bob sends to Alice with
bG + aH $bG + H$ where $H$ is a point of smaller order. This is unlike Answer 1 where he replaces it with $H$.
Are these 2 attacks the same or are they different attacks?
Understanding the attack in Answer 1:
Also, after replacing $bG$, this is what the answer says attacker does next to recover Alice's private key $a$
From the answer
(The attacker is) able to find $a\bmod 8$ by inspecting following messages.
By "following messages" does the author mean the messages which Alice & Bob exchange with each other after the DH part of the protocol is finished - i.e in the ideal case without attack, $abG$ would be the symmetric key which Alice & Bob use to exchange messages after DH is done. Is this what the author means by "following messages". i.e. the attacker takes the message encrypted by the derived secret key & then bruteforces it. The reason why I think this bruteforce will be simpler than a full force of the original order is because since $H$ is a point of small order, $aH$ will also be a point of the same small order & hence the attacker would need to try only the smaller number of points of small order to be successful in brute forcing it? Is my understanding correct?
Understanding the attack in Answer 2:
Here the author says
As Bob knows the value $abG$ (he does his half of the second phase honestly), he can compute Alice's shared secret with $h$ different alternatives, and so recover $a \bmod h$.
Here the author doesn't explicitly say he uses the messages encrypted by the derived shared secret to find Alice's private key - but I am going to assume that's what is the attack.
Since here the secret key generated by Alice would be $a(bG+H)$ - It's not clear to me whether this would also be a point of smaller order which could be bruteforced like the 1st one. Could someone explain the attack here?
Am I on the right track re understanding the cofactor attack?
Regarding Answer 1: yes, you understood it correctly. In practice the attacker will have to brute-force the key of the messages sent by Alice. But it happens often that this bruteforce of the exchanged messages is omitted and implicitly assumed.
Regarding Answer 2: First of all there is a mistake in the question. You wrote
Bob sends to Alice with $bG+aH$
but in fact $a$ is unknown to Bob, $a$ is the final target of the attack. Bob just sends $bG+H$, as correctly written in answer 2.
Now, Bob has computed $bG$, has sent $bG+H$ and has received $aG$ from Alice.
Alice computes the shared secret as $a(bG+H)=abG+aH$ and uses a key derived from it to encrypt her messages.
Bob computes $b(aG)=abG$, now it bruteforce the messages sent from Alice using keys derived from $abG+iH$ for all $i < h$ where $h$ is the cofactor. The one decrypting correctly will give Bob the value of $a \pmod h$.
As you can see the bruteforce effort is the same as in the Answer 1 case.
External links referenced by this document: