The entire idea of the sequence of books prior to the emergence of the codex is a problem to my mind still unresolved. Three ancient testimonies known to me, however, do speak in terms of “order”: b. B. Bathra 14b (seder), the canon list of Melito of Sardis (apud Eus. Hist. eccl. 4.26.12–14; τάξις), and Jerome’s Prologus galeatus (ordo). They all would seem to be presenting what should be the definitive order for the HB/OT, and they all give a different order. For discussion, see N. M. Sarna, “The Order of the Books,” in Studies in Jewish Bibliography, History and Literature in Honor of I. Edward Kiev, ed. Charles Berlin (New York: Ktav, 1971), 407–13, updated as “Ancient Libraries and the Ordering of the Biblical Books,” in Studies in Biblical Interpretation (Philadelphia: JPS, 2000), 53–66 (arrangement in archives/library); Sid Z. Leiman, The Canonization of Hebrew Scripture: The Talmudic and Midrashic Evidence (Hamden, Conn.: Archon Books, 1976), 162 n. 258, 202 n. 644 (writing multiple books on a scroll); Barton, Oracles of God, 82–91 (no significance); Beckwith, Old Testament Canon, 181–234; Sailhamer, Meaning of the Pentateuch, 211, 216 (mental construct); Philip S. Alexander, “The Formation of the Biblical Canon in Rabbinic Judaism,” in Alexander and Kaestli, Canon of Scripture, 57–80 (75–76); Steinberg, Ketuvim, 86–87.As is often pointed out (see, e.g., Steinberg), multi-volume works certainly can have an order across volumes, and this could provide an analogy to these ancient testimonia regarding the order of the biblical books in a society before the domination of the codex (or at least before Jews adopted it as standard for the Bible). But this analogy doesn't seem all that helpful to me, because "order" in multi-volume works can, in fact, mean different things depending on the work. For a multi-volume novel like The Lord of the Rings, "order" means that you should not start reading with the third volume, but with the first, so that you can read the story from beginning to end. This is sometimes the meaning of "order" used to interpret the ancient references to an "order" of the biblical books, such that Baba Bathra, for example, intends that you should start reading the Ketuvim from Psalms and go straight through until you finish with Chronicles.
I have multiple problems with this idea. (1) Who can the Rabbis have expected to read the Ketuvim in this manner, and in what context? The Ketuvim were not read regularly in the synagogue. Can the baraita have been indicating the order the Ketuvim are to be studied in the Beit ha-Midrash? Surely it cannot mean the order in which one is to read the Ketuvim at home. Who would have owned all these scrolls? (I focus on the Ketuvim because it as a collection gives the appearance of less order than do the Torah or Prophets, both of which largely follow a historical order--though not for the Latter Prophets--and because the Torah and Prophets have a regular place in the synagogue liturgy, and so "order" for them might be interpreted to mean the order in which they are to be read liturgically.)
(2) Is there any evidence that anyone ever read the Ketuvim in this order? Is there any evidence that anyone ever read the Ketuvim in any kind of order? Did anyone ever even have a thought to reading the Ketuvim in some sort of order?
(3) The Bible lends itself readily to non-ordered reading (similar to an anthology of poems or of essays). I think this is absolutely the way most Christians today approach the Bible. The exception to this is if someone wants to read the entire Bible in a year, or some similar reading plan, that will sometimes (not always) involve reading straight through from Genesis to Revelation. But outside of this context, personal Bible study often has no thought to order. One might read all the way through Psalms, and then go to Matthew, and then Leviticus, etc. Or one might read a Psalm in the morning, and then a Gospel passage later, or some such. Non-liturgical worship services often feature passages of the Bible according to the whim of the preacher or worship leader. But note that all of those involved in these non-ordered readings of the Bible would say that the Bible has a definite order, and some of them would be able to recite that order almost as easily as reciting the ABCs. It's just that they don't think of that order as intended as a guide to reading. (Yes, they could be wrong, but that's what needs to be proved. I simply want to point out that the connection between "order" and "order for reading" is not so obvious to everyone.)
(4) Finally (for now), there are different ways of thinking about the order of multi-volume works, as I noted before. The Lord of the Rings illustrates one way of thinking about order, but the Anchor Bible Dictionary illustrates another. Now, there is a definite order of volumes for ABD, so that if you want to cite any article you must include not only the page number but also the volume number. But the order of ABD is not intended as an order of reading, and no one would think that the proper way of approaching ABD is to start with vol. 1 and read straight through to the end. If this is the analogy used to understand the ancient references to an order of biblical books, I think we're closer to Sarna's understanding, for whom "order" meant something like "place on a library shelf."
Now, I'm not at all convinced that Sarna is right either, mostly because of a lack of evidence. And so, as I said above, this is an issue still unresolved in my mind. I am interested in exploring the possibility, though, that the references to order are simply for the purposes of memorization, and nothing more. What I mean is, what if "order" of the biblical books is like the "order" of the ABCs. Maybe I'm missing something, but it doesn't seem to me that the order of the ABCs has any significance other than that we had to put them in some order so that we could memorize them without fear that we'd leave a letter out. (For example, if you try to name all 50 states of the USA, you're going to need to use some sort of order or you're likely to omit some. It's probably easiest to use alphabetic order as anything, and that way you know you hit them all.) I think this is how the order of the Bible functions today for Christians. Some scholars have tried to give a significance to the order of the Christian canon (e.g. Marvin Sweeney), but as James Barr astutely noted (in Concept of Biblical Theology, 307-9), and I'm paragraphsing--"I didn't learn that in Sunday school." Christians just do not think about the order of their Bibles as all that significant for reading or interpretation, and I'm not sure that they're wrong for that. But when the order does become important is when they try to list off all 66 books of the Bible. Get one out of order, and it's like putting T before S. It's just not right.
I might also point out--for those who like to see some great significance that the Christian OT ends with Malachi--that this arrangement is actually rather late, widely attested only from the sixth, seventh, or eighth centuries. Earlier, the Twelve were often listed as the first of the Prophets in Christian lists and manuscripts, or, at least, at some position other than last.
Obviously my ideas are not fully developed. I'm just thinking out loud. Maybe the future will bring with it more clarity on this issue.