an icepick through the keyhole

That was what they feared. The reality of course we'll never know. Although there were instances of freed slaves going back to kill their former slave owners, there aren't that many reports of that happening. I like the story of a former slave who, after enlisting in the Union Army, recognized his former master being led away as a prisoner of war, and said, "Hello massa... bottom rail on top now."