Newer BIND versions (9.4 and newer) have the "allow-query-cache" option
which is used to specify which hosts are allowed to get answers from the
cache. (I don't think you are seeing recursion, but just what is already
in your cache.)

With older versions, a workaround is to have a default allow-query for
just your local networks (like your allow-recursion) in the options and
then open up allow-query { any; }; just within your specific zone
statements.