You’re not really comfortable with carrying around a weapon. But hey! Who says a knife has to be a weapon?! It’s a great survival tool! You can cut stuff with it and-… Well, you don’t really have much in the way of survival skills. You doubt Diamond Tiara or Silver Spoon do either. Maybe you could find somepony who does?
By cutting an empty paper tower roll and using some a bunch of tape to reshape it, you craft a Portable Knife Holder! You combine that with a Kitchen Knife to make a Safe-For-Travel Knife and put that in your bag! You might not have any survival skills, but you’re as crafty as a slice of cheese!
… That… That probably would have been funnier if Kraft cheese existed in your world.