> backwards compatibility, which semver is all about

It's not about maintaining backwards compatibility. It's about making it explicit when you break compatibility. Whether you do or not is completely up to you - semver doesn't care.