Depends entirely on the context. In a social setting, of course, conversation is expected and it would be rude to start reading. But in most other situations, other people are not conversation slaves and it is rude to expect them to drop whatever they are doing and act as entertainment. People should not have to lock themselves away in remote, quiet corners in order to read.