Couldn't you make seperate EXEs for the plugins and have the main program send info to them and then receive return info. That way the main program really doesn't wory about how the algorithm works just that it returns a usable output.
That's basically what a DLL is.
Plugins are generally done either through DLLs, or through a script language, which basically acts like a DLL anyway except it doesn't require the user to have the tools for building a DLL (although scripts are slower).
Either way, you need to come up with a common interface, which all plugins use and your application can rely on. For example, a very simple interface would be to require the DLL to have one function which is called: "Encrypt:String(dat:String)". Your app would then load plugin X, get the "Encrypt" function, and call it.
Usually you'll have a slightly more advanced system though to organize the plugins. For example you might require that all DLLs have a "GetEncryptionType:String()" function, so you can identify the type of encryption the DLL implements, and catalog and add it to a list of supported encryption methods in your actual application.
I'm not sure exactly how you'd do this in BlitzMax, but there should be commands for loading DLLs and accessing the functions it contains. What you'll probably want to do is load all DLLs within a "Plugins" folder and add them to a list of available encryption techniques.