I had two "ornamental" pepper plants that had edible peppers, they were about 1/2 inch. The plant bloomed white flowers and when they died there'd be peppers growing from them.
Some stupid spider aphids killed the plants, I tried to save the plants but they died.
I miss the plants, and have never seen them sold in my neck of the woods again.
I don't know what specific species they were, but their peppers were delicious.